about me
play

About Me > Eduardo Silva Github & Twitter: @edsiper Personal - PowerPoint PPT Presentation

Jun 04, 2023 •436 likes •970 views

About Me > Eduardo Silva Github & Twitter: @edsiper Personal Blog : http://edsiper.linuxchile.cl > Treasure Data, Inc. Open Source Engineer Fluentd / FluentBit > Projects Monkey Server / monkey-project.com Duda

  2. About Me > Eduardo Silva ● Github & Twitter: @edsiper ● Personal Blog : http://edsiper.linuxchile.cl > Treasure Data, Inc. ● Open Source Engineer Fluentd / FluentBit > Projects ● Monkey Server / monkey-project.com ● Duda I/O / duda.io

  3. Logging

  4. Logging Matters Pros ● Applications status ● Debugging ● General information about anomalies: errors ● Troubleshooting / Support ● Local or Remote (network)

  5. Logging Matters From a business point of view ● Input Data → Analytics ● User interactions / behaviors ● Improvements

  6. Assumptions

  7. Logging Matters Assumptions ● “I have enough disk space ” ● “I/O operations will not block ” ● “It's fast to write a simple message” ● “Log formats are human readable ” ● “My logging mechanism scale ”

  8. Logging Matters Assumptions B asically, yeah.. it should work.

  9. Concerns

  10. Logging Matters Concerns ● Logs increase = data increase ● Message format get more complex ● Did the Kernel flush the buffers ? ( sync (2)) ● Multi-thread application ?, Locking ? ● M ultiple Applications = M ultiple Logs

  11. Logging Matters Concerns If... Multiple Applications = Multiple logs Multiple Host x Multiple Applications = ???

  12. OK...so: 1. Logging Matters 2. It's really beneficial 3. ??? but...

  13. It needs to be done right.

  14. Logging Common Sources / Inputs ● Applications Logs • Apache • Nginx • Syslog (-ng) ● Custom applications / Languages C , Ruby, Python, PHP, Perl, NodeJS, Java...

  15. How to parse/store multiple data inputs ? note: performance matters!

  17. Fluentd is an open source data collector It let's you unify the data collection for a better use and understanding of data.

  18. Fluentd Highlights ● High Performance ● Built-in Reliability ● Structured Information/Logs ● Plug gable Architecture ● More than 300 plugins (input/filtering/output)

  19. Fluentd Before

  20. Fluentd After

  22. Configuration

  24. Who uses Fluentd in Production ?

  26. Logging and other use cases

  27. We collect Logging and other 500.000 events per second! use cases

  28. Logging and other use cases

  29. Logging More use cases We talked mostly about Server applications, let's talk now about other contexts: ● Internet of Things ( IoT ) ● Embedded Applications ● OS Containers ( Docker ) The fun continues ;)

  30. Internet of Things (IoT)

  31. Internet of Things Facts ● IoT will grow to many billions of devices over the next decade. ● Now it's about device to device connectivity. ● Different frameworks and protocols are emerging. ● It needs Logging .

  32. Internet of Things Frameworks In order to solve the device to device connectivity and proper messaging, multiple framework / stacks are emerging, all of them backed by different companies and vendors. The most remarkable at the moment are: Brillo, by Google Just announced, no logo..sorry!

  33. Internet of Things Logging ● How do we approach data collection for IoT ? ● How can we improve our IoT based product if we have data from thousand of devices with multiple sensors, meaning many inputs ?

  34. Embedded

  35. Embedded Importance of the Embedded Market ● Embedded devices are everywhere: ● Medical Devices ● Automotive ● Home Automation ● Many...

  36. Fluentd for Embedded ?

  38. Fluent Bit Objective Open Source data collection tool for Embedded Linux: ● Services ● Sensors / Signals / Radios ● Operating System Info ● Automotive / Telematics

  39. Fluent Bit Requirements An Embedded environment requires special handling, specifically on performance and resources utilization, for hence it needs to be: ● Lightweight ● Written in C Language. ● Customizable, pluggable architecture. ● Integrate with Fluentd

  40. Fluent Bit Requirements ● Support custom Inputs ● Support custom Outputs ● Use Binary Serialization (MessagePack) ● Config urable (though file system files)

  41. Features

  42. Fluent Bit Features ● C ollection & D istribution. ● Built-in system metrics . ● C API for Developers (WIP). ● Integration with third party services. ● Open Source / Apache License v2.0

  43. Fluentd Integration

  44. Fluent Bit Integration

  45. Direct Output

  46. Fluent Bit Direct Output

  47. Examples

  48. Fluent Bit Built-in Metrics: CPU usage

  49. Fluent Bit Kernel Log Messages

  50. Fluent Bit Configuration Example

  51. Roadmap

  52. Fluent Bit Roadmap ● Library mode. ● Support a stock of sensors (inputs). ● HTTP input. ● Release first stable version, work in process! ● Documentation.

  53. Thanks! http:// fluentd.org http:// fluentbit.io github.com/ fluent/fluentd We are Hiring! github.com/ fluent/fluent-bit Eduardo Silva @edsiper / eduardo@treasuredata.com

Recommend

More recommend