ESB Architecture
- Can be defined by next feautes:
  - Monitoring of services/messages passed between them
  - wire Protocol bridge between HTTP, AMQP, SOAP, gRPC, CVS in Filesystem,...
  - Scheduling, mapping, QoS management, error handling, ..
  - Data transformation
  - Data pipelines
  - Mule, JBoss Fuse (Camel + "etc..."), BizTalk, Apache ServiceMix, ...
REF: https://en.wikipedia.org/wiki/Enterprise_service_bus#/media/File:ESB_Component_Hive.png
    ^   Special App. Services
    |
E   |   Process Automation                 BPEL, Workflow
n   |
t   |   Application Adapters               RFC, BABI, IDoc, XML-RPC, ...
e m |
r e |   Application Data Consolidation     MDM, OSCo, ...
p s |
r s |   Application Data Mapping           EDI, B2B
i a |   _______________________________
s g |   Business Application Monitoring
e e |   _______________________________
    |   Traffic Monitoring Cockpit
S c |
e h |   Special Message Services           Ex. Test Tools
r a |
v n |   Web Services                       WSDL, REST, CGI
i n |
c e |   Protocol Conversion                XML, XSL, DCOM, CORBA
e l |
    |   Message Consolidation              N.N (data locks, multi-submit,...)
B   |
u   |   Message Routing                    XI, WBI, BIZTALK, Seeburger
s   |
    |   Message Service                    MQ Series, MSMQ, ...

Apache NiFi: Data Routing
@[https://nifi.apache.org]
- no-code, .drag-and-drop Web-GUI to build pipelines for
  data route+transform (99% NiFi users will never see a line of code)
- Deployed as a standalone application  (vs framework, api or library)
- Support batch ETL but "prefers" data streams.
- Support for binary data and large files ("multi-GB" video files).
- Complex (and performant) transformations, enrichments and normalisations. 
- out of the box it supports:
  - Kafka, Elastic, HDFS, S3, Postgres, Mongo, etc.
  - Generic sources/endpoints: TCP, HTTP, IMAP, ...

- scalable directed graphs of data routing/transformation/system mediation logic.
- UI to control design, feedback, monitoring
- natively clustered - it expects (but isn't required) to be deployed on
  multiple hosts that work together as a cluster for 
  performance, availability and redundancy.

- Highly configurable:
  - Loss tolerant vs guaranteed delivery
  - Low latency vs high throughput
  - Dynamic prioritization
  - Flow can be modified at runtime
  - Back pressure
- Data Provenance
  - Track dataflow from beginning to end
- Designed for extension
  - Build your own processors and more
  - Enables rapid development and
    effective testing
- Secure
  - SSL, SSH, HTTPS, encrypted content, etc...
  - Multi-tenant authorization and internal
    authorization/policy management

- DevOps:
@[https://dzone.com/articles/setting-apache-nifi-on-docker-containers]
REFS:
- NiFi+Spark:
@[https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark"]

- Nifi "vs" Cammel: [comparative]
@[https://stackoverflow.com/questions/65625166/apache-camel-vs-apache-nifi]

GraphQL Engine
• Blazing fast, instant realtime GraphQL APIs for DB with fine 
  grained access control, also trigger webhooks on database events.
• POLE POSITION in GitHub for Top 32 Automatic Api Open Source Projects on Github (
  https://awesomeopensource.com/projects/automatic-api

• Ex:
  PostgreSQL data                   GraphQL subscription                      UI (Web, Mobile, ...)

  ORDERS TABLE                       subscription fetchOrder {    ·········─→ event pushed to update UI
  ==============================       orders ( where:
  order_id  payment   dispatched                 {id: {_eq: 'XX_11'}}) {
  ---------+--------+-----------         id
  XX-11     true      true               payment
  ---------+--------+-----------         dispatched
  ...                                }

NoCoDB: DDBB to spreedsheet
@[https://www.nocodb.com/]
• Second position in GitHub for Top 32 Automatic Api Open Source Projects on Github (

https://awesomeopensource.com/projects/automatic-api
• Open Source "Airtable Alternative"
• Turns any database into a smart spreadsheet.

Metabase
https://github.com/metabase/metabase
• simplest way to get business intelligence and analytics from a DDBB to everyone.
• Low-code chart (including geo-map) generation.

Datasette.io
• Spin up a JSON API for any data in minutes. 
• Prototype and prove your ideas without building a custom backend.

Redash:
- "Make your company Data Driven"
- Redash is designed to enable anyone, regardless of the level of 
  technical sophistication, to harness the power of data big and small. 
  SQL users leverage Redash to explore, query, visualize, and share 
  data from any data sources. Their work in turn enables anybody in 
  their organization to use the data. Every day, millions of users at 
  thousands of organizations around the world use Redash to develop 
  insights and make data-driven decisions.
- Supported DDBBs:
  Amazon Athena                     MemSQL
  Amazon DynamoDB                   Microsoft Azure Data Warehouse / Synapse
  Amazon Redshift                   Microsoft Azure SQL Database
  Axibase Time Series Database      Microsoft SQL Server
  Cassandra                         MongoDB
  ClickHouse                        MySQL
  CockroachDB                       Oracle
  CSV                               PostgreSQL
  Databricks (Apache Spark)         Presto
  DB2 by IBM                        Prometheus
  Druid                             Python
  Elasticsearch                     Qubole
  Google Analytics                  Rockset
  Google BigQuery                   Salesforce
  Google Spreadsheets               ScyllaDB
  Graphite                          Shell Scripts
  Greenplum                         Snowflake
  Hive                              SQLite
  Impala                            TiDB
  InfluxDB                          TreasureData
  JIRA                              Vertica
  JSON                              Yandex AppMetrrica
  Apache Kylin                      Yandex Metrica
  OmniSciDB (Formerly MapD)

Business Data charts
@[https://www.forbes.com/sites/bernardmarr/2017/07/20/the-7-best-data-visualization-tools-in-2017/#643a48726c30]

Apache Superset: [TODO]
@[https://www.phoronix.com/scan.php?page=news_item&px=Apache-Superset-Top-Level]
Superset is the project's big data visualization and business 
intelligence web solution. Apache Superset allows for big data 
exploration and visualization with data from a variety of databases 
ranging from SQLite and MySQL to Amazon Redshift, Google BigQuery, 
Snowflake, Oracle Database, IBM DB2, and a variety of other 
compatible data sources. 
Python based.

Superset is also cloud-native in the sense that it is flexible and 
lets you choose the:
- web server (Gunicorn, Nginx, Apache),
- metadata database engine (MySQL, Postgres, MariaDB, etc),
- message queue (Redis, RabbitMQ, SQS, etc),
- results backend (S3, Redis, Memcached, etc),
- caching layer (Memcached, Redis, etc),

Superset also works well with services like NewRelic, StatsD and 
DataDog.

Low-Level Data charts
|ºKibanaº (TODO)
|- "A Picture's Worth a Thousand Log Lines"
|- visualize (Elasticsearch) data and navigate
|  the Elastic Stack, learning  understanding
|  the impact rain might have on your quarterly
|  numbers

ºGrafanaº (TODO)
@[https://grafana.com/]
- time series analytics
- Integrates with Prometheus queries
@[https://logz.io/blog/grafana-vs-kibana/]
- 7.0+ includes new plugin architecture, visualizations,
  transformations, native trace support, ... 
https://grafana.com/blog/2020/05/18/grafana-v7.0-released-new-plugin-architecture-visualizations-transformations-native-trace-support-and-more/ 

TOGAF
@[https://en.wikipedia.org/wiki/The_Open_Group_Architecture_Framework]
The Open Group Architecture Framework (TOGAF) is a framework for enterprise
architecture that provides an approach for designing, planning, implementing,
and governing an enterprise information technology architecture.[2] TOGAF is
a high level approach to design. It is typically modeled at four levels:
Business, Application, Data, and Technology. It relies heavily on
modularization, standardization, and already existing, proven technologies and
products.

TOGAF was developed starting 1995 by The Open Group, based on DoD's TAFIM. As
of 2016, The Open Group claims that TOGAF is employed by 80% of Global 50
companies and 60% of Fortune 500 companies.[3]

Summary from @[https://www.redhat.com/architect/TOGAF-importance]
• organization created in 1996. 850-global members as of 2021-11
  advancing in technology standards services, certifications,
  research, and training.
• The Open Group is behind many world-class architectural and digital 
  standards, including:
  - Open Agile Architecture (O-AA) Standard
  - ArchiMate Modeling Language
  - Certifying body for the Unix trademark.
  - developing and management of the 
   ºOpen Group Architecture Framework (TOGAF)º
    key industry-standard for enterprise architecture.
    ( more than 110,000 TOGAF certifications issued in 155 countries) 

 "... organizations are finding the need to pull these disparate 
  projects together and are turning to TOGAF Standard and enterprise 
  architecture as the most proven and reliable methodology of achieving 
  this. Understanding and managing that big picture of an enterprise 
  has always been at the heart of TOGAF Standard, and it will be 
  exciting to see how practitioners apply it to this next stage of 
  digitalization ...."

  "... What TOGAF doesn't do is tell you how to do your job: 
   TOGAF doesn't give us a playbook, instead, it realigns the way we 
   solve problems into a common framework, approach, and a language that 
   can be communicated across a team of architects, a practice, an 
   industry, and across our profession."
   
   "...TOGAF works because it provides a framework to identify and promote 
    validated patterns across an evolving set of customer and 
    industry-specific architectures that are then evolved into deployable 
    solution blueprints."

Kogito
@[http://kie.org/]
- Kogito == Drools + jBPM + OptaPlanner
BºDroolsº
  - Drools is a business rule management system with a 
    forward-chaining and backward-chaining inference based rules engine, 
    allowing fast and reliable evaluation of business rules and complex 
    event processing. 
    Also use in other projects like https://www.shopizer.com/, and OOSS
    java ecommerce solution based on Spring/Hibernate/Drools/Elastic.

BºjBPMº
@[https://www.jbpm.org/]
  - toolkit for building business applications to help automate business processes and decisions.
  - jBPM originates from BPM (Business Process Management) but it has 
    evolved to enable users to pick their own path in business 
    automation. It provides various capabilities that simplify and 
    externalize business logic into reusable assets such as cases, 
    processes, decision tables and more.
    - business processes (BPMN2)
    - case management (BPMN2 and CMMN)
    - decision management (DMN)
    - business rules (DRL)
    - business optimisation (Solver)
  
  - jBPM can be used as standalone service or embedded in custom 
    service. It does not mandate any of the frameworks to be used, it can 
    be successfully used in
    - traditional JEE applications - war/ear deployments
    - SpringBoot, Thorntail, ... deployments
    - standalone java programs

BºOptaPlannerº [optimization]
- OptaPlanner is an AIºconstraint solverº. It optimizes planning and 
  scheduling problems, such as the Vehicle Routing Problem, Employee 
  Rostering, Maintenance Scheduling, Task Assignment, School 
  Timetabling, Cloud Optimization, Conference Scheduling, Job Shop 
  Scheduling, Bin Packing and many more. Every organization faces such 
  challenges: assign a limited set of constrained resources (employees, 
  assets, time and/or money) to provide products or services. 
  OptaPlanner delivers more efficient plans, which reduce costs and 
  improve service quality.

Summary
• Messaging represents a layer down ESB in some cases.
  Messaging architectures focus on reliable delivery of messages of different nature/timelife.
  On top of them, ESB add data management/processing/transformation.
• In general a message in message driven architectures represents some information that
 *MUST* be processed, while an event in event stream architecture ("Kafka", "Pulse", fast 
  distributed queues) represent a more generic information that MUST or MUST not be processed.
    Event streams can most of the time be used to build a message stream architecture on top
  and ussually the have better real-time performance and/or scalability (big data) performance.
    On the other side conventional messaging architectures focus on warranting the proper
  delivery and consumption of messages.
    For example, messages triggering transactions representing "money" that do not require 
  big-data scalation are better deal with message archictectures than general event stream 
  architectures.

- Message classification by consumption model:
 ┌──────────────────────────────────────────────────────────────────────────────────────────┐
 │ publish/  │ Publishers send messages (commands) to be consumed-and-forgotten.            │
 │ subscribe │ Events cannot be replayed after being received.                              │
 │           │ New consumers will not be able to retrieve past events.                      │
 │           │ Messages are ussually interpreted and processed in the same way              │
 │           │ by all clients. They behave like a sort of async-and-remote function call.   │
 │           │ where subscribing clients acts as the called function.                       │
 │           │ subscribers are decoupled in time, but coupled in logic.                     │
 │──────────────────────────────────────────────────────────────────────────────────────────│
 │ Event     │ Publishers write new events (source-of-true)in order to a long-lasting log.  │
 │ streaming │ Consumers don't need to subscribe, and they can read from any part of the    │
 │           │ even stream log, allowing to replay reads.                                   │
 │           │ Different clients can interpret and process event stream logs differently    │
 │           │ and for different purposes.                                                  │
 │           │ subscribers are decoupled in time and in logic.                              │
 └──────────────────────────────────────────────────────────────────────────────────────────┘


- Messages classification by message expiration-time validity:
 ┌────────────────────────────────────────────────────────────────────┐
 │MESSAGE VALID           │ MESSAGIN SYSTEM NEEDS │ Consumption model │
 │────────────────────────│───────────────────────│───────────────────│
 │for a short time        │ fast delivery         │ publish/subscribe │ 
 │────────────────────────│───────────────────────│───────────────────│
 │until consumed          │ persistence           │ publish/subscribe │
 │────────────────────────│───────────────────────│───────────────────│
 │for repeated consumption│ persistence and index │ event streaming   │
 └────────────────────────────────────────────────────────────────────┘

- Message classification by semmantics:
  ┌───────────────────────────────────────────────────────────────────────────┐
  │NATURE          │                            │                             │
  │────────────────│────────────────────────────│─────────────────────────────│
  │Command/Order   │·leads state change         │· Async resilient alternative│
  │                │·Requires response/result   │  to sync HTTP PUT/POST      │
  │────────────────│────────────────────────────│─────────────────────────────│
  │Query           │·Requires response/result   │· Async resilient alternative│
  │                │ sync/async(fire and forget)│  to sync HTTP PUT/POST      │
  │────────────────│────────────────────────────│─────────────────────────────│
  │Event           │·Doesn't require response   │· Used in Cmd-Query Responsa-│
  │                │                            │  bility Segregation         │
  └───────────────────────────────────────────────────────────────────────────┘

   Ex messaging system:
   SYSTEM           BEST FOR    SEMANTICS    WORST FOR
  ┌───────────────────────────────────────────────────────┐
  │MQ "Classic  │ ·Durable     │·Command │ ·Replayable    │
  │Broker"      │ ·Volatile    │·event   │                │
  │─────────────│──────────────│─────────│────────────────│
  │KAFKA        │ ·Replayable  │·Event   │ ·Volatile      │
  │             │ ·Scalability │         │ ·light hardware│
  │             │ ·Decoupled   │         │                │
  │             │  components  │         │                │
  │─────────────│──────────────│─────────│────────────────│
  │Qpid         │ ·Volatile    │·Command │ ·Durable       │
  │             │              │·Query   │ ·Replayable    │
  └───────────────────────────────────────────────────────┘
 ┌───────────────────────────────────────────────────────────────────────────────────────────────┐
 │  Working   │ Description                │ PROS                      │ CONS                    │
 │  Model     │                            │                           │                         │
 │───────────────────────────────────────────────────────────────────────────────────────────────│
 │  queuing   │ pool of consumers may read │ allows to split processing│ queues aren't           │
 │            │ from server and each record│ over multiple consumers,  │ multi─subscriber—once   │
 │            │ goes to one of them        │ scaling with easy         │ one process reads the   │
 │            │                            │                           │ data it's gone.         │
 │───────────────────────────────────────────────────────────────────────────────────────────────│
 │  publish/  │ record broadcast           │ broadcast data to         │ no way of scaling       │
 │  subscribe │ to all consumers.          │ multiple processes        │ since every message     │
 │            │                            │                           │ goes to every subscriber│
 └───────────────────────────────────────────────────────────────────────────────────────────────┘


Bº##################º
Bº# Message Queues #º
Bº##################º
  Defined by
  - message oriented architecture
  - Persistence (or durability until comsuption)
  - queuing
  - Routing: point-to-point / publish-and-subscribe
  - No processing/transformation of message/data

Bº########º
Bº# AMQP #º (RabbitMQ, APache ActiveMQ, Qpid, Solace, ...)
Bº########º
  @[https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol]
    - Open standard network protocol
  • binary wire protocol designed as open replacement for existing 
    proprietary messaging middleware. 
  • strong enterprise adoption. (JP Morgan claims to process 
    1+ billion messages a day. Also used by NASA, Google,...) 


    - Often compared to JMS:
      - JMS defines API interfaces, AMQP defines network protocol
      - JMS has no requirement for how messages are formed and
        transmitted and thus every JMS broker can implement the
        messages in a different (incompatible) format.
        AMQP publishes its specifications in a downloadable XML format,
        allowing library maintainers to generate APIs driven by
        the specs while also automating construction of algorithms to
        marshal and demarshal messages.
    - brokers implementations supporting it:
      
      @[https://www.amqp.org/about/examples]
      - Apache Qpid (TODO)
      @[http://qpid.apache.org/]
      - INETCO's AMQP protocol analyzer
      @[http://www.inetco.com/resource-library/technology-amqp/]
      - JORAM  JMS + AMPQ
      @[http://joram.ow2.org/] 100% pure Java implementation of JMS
      - Kaazing's AMQPºWeb Clientº
      @[http://kaazing.net/index.html]
      - Azure Service Bus+ AMPQ
  -RºWhat's wrong with AMQP?º. Extracted from
   @[https://news.ycombinator.com/item?id=1657574]
    """ ... My answer, as someone who recentlyºswitched an AMQP backend º
     ºto using redisºfor queues and pub/sub broadcast: AMQP is wrong the   [comparative]
      same way RDBMSes or Java are "wrong." It's too heavy, too slow, or 
      too complex for a great-many use cases...
    Bº  The truth is that something with a radically reduced feature set, º
    Bºthat's simpler to use and understand, is really all that's needed toº 
    Bºmeet a whole lot of "real world" use cases.                         º

  - @[http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-amqp-overview/]
  - JBoss A-MQ, built from @[http://qpid.apache.org/]
  @[http://www.redhat.com/en/technologies/jboss-middleware/amq]
  - IBM MQLight
  @[https://developer.ibm.com/messaging/mq-light/]
  - StormMQ
  @[http://stormmq.com/]
    a cloud hosted messaging service based on AMQP
  - RabbitMQ (by VMware Inc)
  @[http://www.rabbitmq.com/];
    also supported by SpringSource
  ...
  - Java JMS

• See also:
  ·@[../WebTechnologies/map.html#asyncapi_summary] standard to model   [standard] 
    event-driven async APIs.
  · "The many meanings of event-driven architecture,"
    @[https://www.youtube.com/watch?v=STKCRSUsyP0] (By Martin Fowler)


Message Brokers
- Routing
- (De-)Multiplexing of messages from/into multiple messages to different recipients
- Durability
- Transformation (translation of message between formats)
- "things usually get blurry - many solutions are both (message queue and message
  broker) - for example RabbitMQ or QDB.  Samples for message queues are
  Gearman, IronMQ, JMS, SQS or MSMQ."
  Message broker examples are, Qpid, Open AMQ or ActiveMQ.
- Kafka can also be used as message broker but is not its main intention

ActiveMQ
• Based on JAVA
Bº######################
Bº# ActiveMQ "Classic" #
Bº######################
  endlessly pluggable architecture serving many generations of 
  applications.
  · Web interface.
  · JMS 1.1 with full client implementation including JNDI
    (RºNo JMS 2.0 supportº)
  ·ºHigh availabilityºusing shared storage
  · Familiar JMS-based addressing model
  · "Network of brokers" for distributing load
  · KahaDB+JDBC options for persistence
  ·BºQUICK DEVELOPMENT SETUP:º
     → Setup Java/$JAVA_HOME
       → Download ActiveMQ Classic 
         → Execute (apache-activemq-5.16.2/bin/)wrapper
           → Open http://localhost:8161/admin/index.jsp
             → "Queues" tab → "Create" button
               DONE!!!
  
Bº####################
Bº# ActiveMQ Artemis #
Bº####################
• Multiprotocol("polyglot") broker (AMQP, OpenWire, MQTT, STOMP, HTTP, ...)
• Single / multi-cluster.
• Java based
• next generation High-performance, non-blocking architecture  
  · Web interface.
  · JMS 1.1, 2.0 with full client implementation including JNDI
  ·ºHigh availabilityºusing shared storage or network replication
  · Simple + powerful protocol agnostic addressing model
  · Flexible clustering for distributing load
  · Advanced journal implementations for low-latency persistence
    as well as JDBC
  · High feature parity with ActiveMQ "Classic" to ease migration
  · Very good and free documentation:
  @[https://activemq.apache.org/components/artemis/documentation/latest/book.pdf]


RabbitMQ 
• Erlang Alternative. Quite popular since it was used as AMQP 
  ref.implementation.
@-[https://www.rabbitmq.com/getstarted.html]
• Based on Erlang (Rºvery reduce developer communityº)
• Multiprotocol("polyglot") broker (AMQP, MQTT, STOMP, ...)
  e.g.: RabbitMQ Web Stomp allows for example to expose messaging in a browser
        through websockets.

• P: Producer, C: consumer

  P →  │││queue_name │││ ──→ C                                Work queues
                         └─→ C                               ------------------

            ┌→ │││queue1 │││ ──→ C
  P →  │X│ ─┤                                                 Publish/Subscribe
            └→ │││queue2 │││ ──→ C                            ------------------



            ┌→ (error) │││queue1 │││ ──→ C
  P →  │X│ ─┤                                                 Routing
        ^   │       info                                      -------
        |   └───────error──→ │││queue2 │││ ──→ C              
                    warning 
     type=direct


            ┌→ *.orange.* → │││queue1 │││ ──→ C
  P →  │X│ ─┤                                                 Topics
        ^   │                                                 ------
        |   └─ *.*.rabbit → │││queue2 │││ ──→ C 
               lazy.*
     type=topic


  C ─→ Request:            ──→ │││ request  │││ ──→  Server   Async RPC
  ^    reply_to=amqp.gen...                           │       ---------
  │    correlation_id=abc                             │
  │                                                   │
  └─── Reply               ─── │││ response │││ ──────┘ 
         correlation_id=abc

See @[#activemq_summary]

OpenHFT: microSec Messaging storing to disk
@[https://github.com/OpenHFT/]
   - Chronicle-Queue: Micro second messaging that stores everything to disk
   - Chronicle-Accelerate: HFT meets Blockchain in Java platform
         XCL is a new cryptocurrency project that, learning from the previous
         Blockchain implementations, aims to solve the issues limiting adoption
         by building an entirely new protocol that can scale to millions of
         transactions per second, delivering consistent sub-second latency. Our
         platform will leverage AI to control volatility and liquidity, require
         low energy and simplify compliance with integrated KYC and AML support.

         The XCL platform combines low latencies (sub millisecond), IoT transaction
         rates (millions/s), open source AI volatility controls and blockchain for
         transfer of value and exchange of value for virtual fiat and crypto
         currencies. This system could be extended to other asset classes such as
         securities and fixed income. It uses a federated services model and
         regionalized payment systems making it more scalable than a blockchain
         which requires global consensus.

         The platform makes use of Chronicle-Salt for encryption and Chronicle-Bytes.

         on Chronicle Core’s direct memory and OS system call access.



   - Chronicle-Logger: A sub microsecond java logger, supporting standard logging
                      APIs such as Slf & Log4J

CQRS
REF: @[https://www.redhat.com/architect/pros-and-cons-cqrs]
• CQRS stands for Command Query Responsibility Segregation 
• The data-processing architecture pattern separates a service's write
  tasks from its read tasks:
• Separating write activity from ready activities allows you to use the 
  best database technology for the task at hand, for example, a stream-event
  engine (Kafka/Pulse/...) for writing and a non-SQL database for reading.
• While reading and writing to the same database is acceptable for 
  small applications, distributed applications operating at web-scale 
  require a segmented approach.
• Typically there's more read activity than write activity. Also,
  read activity is immutable. Thus, replicas dedicated to reading data
   can be spread out over a variety of geolocations.
• This approach allows users to get the data that closest to them 
  resulting in a more efficient and scalable solution.

OpenTelemetry
@[https://opentelemetry.io/]
- Emerging suite of standards for (Cloud Native) Observability.
- integration with modern observability SaaS providers like
  Honeycomb, Datadog, Stackdriver, ...

• OpenTelemetry Announces Roadmap for Metrics Specification:
@[https://www.infoq.com/news/2021/03/opentelemetry-metrics-roadmap/]
  The OpenTelemetry project announced its roadmap for its metrics 
  specification. The roadmap includes a stable metrics API/SDK, metrics 
  data model and protocol, and compatibility with Prometheus.

Zipkin
@[https://zipkin.io/]

Genesis Distributed Testing
@[https://docs.whiteblock.io/introduction_to_testing.html]
The following are types of tests one can perform on a distributed system:
- Functional Testing is conducted to test whether a system performs as it was 
  specified or in accordance with formal requirements
- Performance Testing tests the reliability and responsiveness of a system 
  under different types of conditions and scenarios
- Penetration Testing tests the system for security vulnerabilities
- End-to-End Testing is used to determine whether a system’s process flow 
  functions as expected
- Fuzzing is used to test how a system responds to unexpected, random, or 
  invalid data or inputs

Genesis is a versatile testing platform designed to automate the tests listed 
above, making it faster and simpler to conduct them on distributed systems 
where it was traditionally difficult to do so. Where Performance, End-to-End, 
and Functional testing comprise the meat of Genesis’ services, other types of 
testing are enabled through the deployment of services and sidecars on the 
platform.

End-to-End tests can be designed by applying exit code checks for process 
completion, success, or failure in tasks and phases, while Performance tests 
can be conducted by analyzing data from tests on Genesis that apply a variety 
of network conditions and combinations thereof. Functional tests can use a 
combination of tasks, phases, supplemental services and sidecars, and network 
conditions, among other tools.

These processes and tools are further described in this documentation.

Jepsen Project
@[https://jepsen.io/]
Jepsen is an effort to improve the safety of distributed databases, queues, 
consensus systems, etc. We maintain an open source software library for systems 
testing, as well as blog posts and conference talks exploring particular 
systems’ failure modes. In each analysis we explore whether the system lives 
up to its documentation’s claims, file new bugs, and suggest recommendations 
for operators.

Jepsen pushes vendors to make accurate claims and test their software 
rigorously, helps users choose databases and queues that fit their needs, and 
teaches engineers how to evaluate distributed systems correctness for 
themselves.

keycloak
@[https://www.keycloak.org/]
@[https://medium.com/keycloak/keycloak-essentials-86254b2f1872]                   [TODO]
@[https://medium.com/keycloak/keycloak-realm-client-configuration-dfd7c8583489]

• See also @[../WebTechnologies/map.html#keycloak_angular_openid]

"""Add authentication to applications and secure services with minimum fuss"""
-  No need to deal with storing users or authenticating users.
- It's all available out of the box.
- Advanced features such as User Federation, Identity Brokering and Social Login.

   Single-Sign On                        LDAP and Active Directory
   Login once to multiple applications   Connect to existing user directories
   Standard Protocols                    Social Login
   OpenID Connect, OAuth 2.0             Easily enable social login
   and SAML 2.0

   Identity Brokering                    Clustering
   OpenID Connect or SAML 2.0 IdPs       For scalability and availability
   High Performance                      Themes
   Lightweight, fast and scalable        Customize look and feel

   Extensible
   Customize through code
   Password Policies
   Customize password policies


- AA using the Keycloak REST API
@[https://developers.redhat.com/blog/2020/11/24/authentication-and-authorization-using-the-keycloak-rest-api/]
@[https://github.com/edwin/java-keycloak-integration]
Ex Use Case:
- centralized ID platform for Indonesia’s Ministry of Education
  single sign-on:
  Identity Roles:
  - Students (IDs)
  - Teachers (IDs)
 - each user have different access and privileges at each school.

BºSETUP:º {
  → Creating Keycloak realm:
    → Add 'education' realm for ministry.
      (click enable and create)
      → go to 'Roles' page:
        → choose tab "Realm Roles".
          → Create two roles:
            - "teacher"
            - "student"
            → (Clients page) click Create to add a client:
              - create "jakarta-school" client and save
              → Go to jakarta-school details page → "Settings tab":
                - Enter next client config:
                  Client ID: jakarta-school
                  Enabled: ON
                  Consent Required: OFF
                BºClient Protocol: openid-connectº
                  Access Type: confidential
                BºStandard Flow Enabled: ONº
                  Impact Flow Enabled: OFF
                BºDirect Access Grants Enabled: ONº
                  Browser Flow: browser            (← Scroll to bottom)
                  Direct Grant Flow: direct grant
                → Go to tab "Roles" tab → Add next Roles:
                  - "create-student-grade"
                  - "view-student-grade"
                  - "view-student-profile"
                  → go to tab "Client Scopes" → Default Client Scopes and   
                    add “roles” and “profile” to Assigned Default Client
                    Scopes.
                    → go to "jakarta-school details" page → Mappers →
                      Create Protocol Mappers: 
                      - Set mappers to display the client roles on the
                        Userinfo API:
                        Name               : roles
                        Mapper Type        : User Realm Role
                        Multivalued        : ON
                        Token Claim Name   : roles
                        Claim JSON Type    : String
                        Add to ID token    : OFF
                        Add to access token: OFF
                        Add to userinfo    : ON
                      → go to "Users page" → "Add user".
                        - create the new users and Save. Ex:
                          Username      : edwin
                        BºEmail         : edwin@redhat.comº
                          First Name    : Edwin
                          Last Name     : M
                          User Enabled  : ON
                        BºEmail Verified: OFFº
                        → Go to tab "Role Mappings" → Client Roles 
                          (for each user in jakarta-school)
                          Check that roles perm. are enabled:
                          "create-student-grade"
                          "view-student-grade"
                          "view-student-profile".
}

BºSpring Boot Java App using Keycloak for AAº
  - pom.xml:
    ˂?xml version="1.0" encoding="UTF-8"?˃
    ...  
        ˂dependencies˃
        ...  
            ˂dependency˃
                ˂groupId˃com.auth0˂/groupId˃
                ˂artifactId˃jwks-rsa˂/artifactId˃
                ˂version˃0.12.0˂/version˃
            ˂/dependency˃
    
            ˂dependency˃
                ˂groupId˃com.auth0˂/groupId˃
                ˂artifactId˃java-jwt˂/artifactId˃
                ˂version˃3.8.3˂/version˃
            ˂/dependency˃
        ˂/dependencies˃
    ˂/project˃

  - Config. properties file:
    server.error.whitelabel.enabled=false
    spring.mvc.favicon.enabled=false
    server.port = 8082
    
    keycloak.client-id=jakarta-school
    keycloak.client-secret=197bc3b4-64b0-452f-9bdb-fcaea0988e90
    keycloak.scope=openid, profile
    keycloak.authorization-grant-type=password
    
    keycloak.authorization-uri=http://localhost:8080/auth/realms/education/protocol/openid-connect/auth
    keycloak.user-info-uri=http://localhost:8080/auth/realms/education/protocol/openid-connect/userinfo
    keycloak.token-uri=http://localhost:8080/auth/realms/education/protocol/openid-connect/token
    keycloak.logout=http://localhost:8080/auth/realms/education/protocol/openid-connect/logout
    keycloak.jwk-set-uri=http://localhost:8080/auth/realms/education/protocol/openid-connect/certs
    keycloak.certs-id=vdaec4Br3ZnRFtZN-pimK9v1eGd3gL2MHu8rQ6M5SiE
    
    logging.level.root=INFO
    
  - Config: Get the Keycloak public certificate ID from:
    Keycloak → Education → Keys → Active (with RSA-generated key selected).

  - Integration code: (several Keycloak APIs are involved).

    - logout API:
      @Value("${keycloak.logout}")
      private String keycloakLogout;

      public void logout(String refreshToken) throws Exception {
        MultiValueMap˂String, String˃ map = new LinkedMultiValueMap˂˃();
        map.add("client_id",clientId);
        map.add("client_secret",clientSecret);
        map.add("refresh_token",refreshToken); // ← RºWARNº: Use refresh 
                                               //   (vs access) token

        HttpEntity˂MultiValueMap˂String, String˃˃ request = new HttpEntity˂˃(map, null);
        restTemplate.postForObject(keycloakLogout, request, String.class);
      }

    - Check whether a bearer token is valid and active or not
      @Value("${keycloak.user-info-uri}")
      private String keycloakUserInfo;
      private String getUserInfo(String token) {
        MultiValueMap˂String, String˃ headers = new LinkedMultiValueMap˂˃();
        headers.add("Authorization", token);
        HttpEntity˂MultiValueMap˂String, String˃˃ request =
          new HttpEntity˂˃(null, headers);
        return restTemplate.postForObject(keycloakUserInfo, request, String.class);
      }

    - Authorization (== "role valid for a given API"?) alternatives:
      - Alt 1. determine what role a bearer token brings by verifying it 
               against Keycloak’s userinfo API
             RºDrawback:º multiple roundtrip request needed
        following response from Keycloak is expected:
        {
            "sub": "ef2cbe43-9748-40e5-aed9-fe981e3082d5",
          Oº"roles": [ "teacher" ],º
            "name": "Edwin M",
            "preferred_username": "edwin",
            "given_name": "Edwin",
            "family_name": "M"
        }

      - Alt 2: decode JWT bearer token and validate it.
        - code needs firt the public key used for signing the token:
          (It requires a round trip but can be cached)
          - Keycloak provides a JWKS endpoint. Ex:
            $ curl -L -X GET 'http://.../auth/realms/education/protocol/openid-connect/certs'
            { "keys": [
                {
                  "kid": "vdaec4Br3ZnRFtZN-pimK9v1eGd3gL2MHu8rQ6M5SiE", ← key id
                  "kty": "RSA",
                  "alg": "RS256",
                  "use": "sig",
                  "n": "4OPCc_LDhU6ADQj7cEgRei4....",← Oºpublic key usedº
                  "e": "AQAB"                          Oºfor decodingº
                }
              ] }
        - A sample decoded JWT token will look like:
          {
            "jti": "85edca8c-a4a6-4a4c-b8c0-356043e7ba7d",
            "exp": 1598079154,
            "nbf": 0,
            "iat": 1598078854,
            "iss": "http://localhost:8080/auth/realms/education",
            "sub": "ef2cbe43-9748-40e5-aed9-fe981e3082d5",
            "typ": "Bearer",
            "azp": "jakarta-school",
            "auth_time": 0,
            "session_state": "f8ab78f8-15ee-403d-8db7-7052a8647c65",
            "acr": "1",
            "realm_access": { "roles": [ "teacher" ] },
            "resource_access": {
              "jakarta-school": {
                "roles": [ "create-student-grade", "view-student-profile",
                  "view-student-grade" ]
              }
            },
            "scope": "profile",
            "name": "Edwin M",
            "preferred_username": "edwin",
            "given_name": "Edwin",
            "family_name": "M"
          }
          Ex JWT validation code:
          @GetMapping("/teacher")
          HashMap teacher(
              @RequestHeader("Authorization") String authHeader) {
                  try {
              DecodedJWT jwt = JWT.decode(
                                 authHeader.replace("Bearer", "").trim());
              Jwk jwk = jwtService.getJwk();
              Algorithm algorithm = Algorithm.RSA256(
                        (RSAPublicKey) jwk.getPublicKey(), null);
              algorithm.verify(jwt);                // ← Bºcheck signatureº

              List˂String˃ roles = ((List)jwt.getClaim("realm_access").
                                    asMap().get("roles"));
              if(!roles.contains("teacher"))        //  ← Bºcheck rolesº
                  throw new Exception("not a teacher role");

              Date expiryDate = jwt.getExpiresAt(); // ← Bºcheck expirationº
              if(expiryDate.before(new Date()))
                  throw new Exception("token is expired");

              return new HashMap() {{ put("role", "teacher"); }}; // validation OK
                  } catch (Exception e) {
              throw new RuntimeException(e);
                  }
          }

FreeIPA
@[https://www.reddit.com/r/linuxadmin/comments/apbjtc/freeipa_groups_and_linux_usernames/]

API Management
@[https://www.datamation.com/applications/top-api-management-tools.html]

Behaviour
Driven
Development
(BDD)
- Cucumber, ...
@[https://cucumber.io/docs/guides/10-minute-tutorial/]

Infra vs App Monitoring
See also @[../JAVA/java_map.html#mdc_summary]
BºInfrastructure Monitoring:º
  - Prometheus + Grafana
    (Alternatives include Monit, Datadog, Nagios, Zabbix, ...)

BºApplication distributed tracing/Monitoring (End-to-End Request Tracing)º
  - Jaeger (OpenTracing compatible), Zipkin, New Relic
    (Alternatives include AppDynamics, Instana, ...) 
  -ºtrack operations inside and across different systemsº.
  - Check for example how an incomming request affected to the web
    server, database, app code, queue system, all presented along 
    a timeline.
  - Especially valuable in distributed systems.
  - It complements logs and metrics:
    - App.request metrics warns about latencies in remote requests
      while local traces just do at local isolated systems. 
    - Logs on each isolated system can explain why such system is "slow".
    - Infra. monitoring warn about scarcity of CPU/memory/storage resources.
  - See also notes on Spring Cloud Sleuth @[../JAVA/java_map.html] that
    offers an interface to different tracing tech.stacks. (Jaeger,...)

  • Related: https://github.com/dapr/dapr 
    Dapr: portable, event-driven, runtime for building distributed
          applications across cloud and edge.  
  
    ...by letting Dapr’s sidecar take care of the complex challenges such 
    as service discovery, message broker integration, encryption, 
    observability, and secret management, you can focus on business logic 
    and keep your code simple.
    ...  usually a developer must add some code to instrument an 
    application for this purpose send collected data to external 
    monitoring tool ...  Having to maintain this code adds another
    burden sometimes requiring understanding the monitoring tools' 
    APIs, additional SDKs ... different cloud providers offer 
    different monitoring solutions.
    • Observability with Dapr:
    • When building an application which leverages Dapr building blocks 
      to perform service-to-service calls and pub/sub messaging, Dapr 
      offers an advantage with respect to distributed tracing. Because this 
      inter-service communication flows through the Dapr sidecar, the 
      sidecar is in a unique position to offload the burden of 
      application-level instrumentation.
    • Distributed tracing
      Dapr can be configured to emit tracing data, and because Dapr does so 
      using widely adopted protocols such as the Zipkin protocol, it can be 
      easily integrated with multiple monitoring backends.

BºLogging How Toº:
  → Start by adding logs and infra monitoring
    → add application monitoring:
      ( requires support from programming languages/libraires, developpers and devOps teams)
      - instrument code (Jaeger)
      - add tracing to infrastructure components 
        - load balancers
      → deploy App tracing system itself. 
        (Ex.: Jaeger server, ...)
 
Bº(Bulk) Log Managementº
  - Elastic Stack
    (Alternative include Graylog, Splunk, Papertrail, ...)

Jaeger (App.Monit)
@[https://www.jaegertracing.io/]
@[https://logz.io/blog/zipkin-vs-jaeger/]
- Support 
- Support for Open Tracing instrumentation libraries
  like @[https://github.com/opentracing-contrib].
- K8s Templates and Operators supported.                          [k8s]
@[https://developers.redhat.com/blog/2019/11/14/tracing-kubernetes-applications-with-jaeger-and-eclipse-che/]
  
- Jaeger addresses next problems:
  - end-to-end distributed transaction monitoring/tracing.
    troubleshooting complex distributed systems.
  - performance and latency optimization.
  - root cause analysis.
  - service dependency analysis.
  - distributed context propagation.

 º"... As on-the-ground microservice practitioners are quickly realizing, theº
 º majority of operational problems that arise when moving to a distributedº
 º architecture are ultimately grounded in two areas: networking andº
 º observability. It is simply an orders of magnitude larger problem to networkº
 º and debug a set of intertwined distributed services versus a singleº
 º monolithic application..."º

  Jaeger Log Data Architecture Schema:
  (Based on OpenTracing)
@[https://www.jaegertracing.io/docs/1.21/architecture/]
@[https://github.com/opentracing/specification/blob/master/specification.md]

  ºSPANº                    ºTRACEº:
   (logical work-unit)       - data/execution path 
   ----------------            through system components
   - operation name            (sort of directed acyclic
   - start time                 graph of spans).
   - duration.
   - Spans may be nested
     and ordered to model 
     causal relationships.

     Unique ID → │A│
     (Context)    └─│B│
                  └─│C│─│D│
                  └········│E│
                   ^^^^^^^^
                   barrier waiting for 
                   │B│,│C│ results
     

     ········→ time line →··············· 
                                          ┐
     ├─────────────  SPAN A ────────────┤ │TRACE
       ├──── SPAN B ───┤                  │
       ├─ SPAN C ─┤                       │
                  ├─ SPAN D ─┤            │
                             ├─ SPAN E ─┤ │
                                          ┘
   
   
• Jaeger Components Schema:

  BºDATA INPUTº
    Application                             jaeger-collector
    └→ App Instrumetation                ┌→ (Go Lang)
       └→ OpenTracing API                ·  + memory-queue ·┐
          └→ Jaeger-client ···→ Jaeger ··┘                  · 
                                agent                       ·
                              (Go lang)                     ·
                                                            ·
                                             Data Store  ←··┘
                                            (Cassandra,
                                             Elastic Search,
                                             Kafka, Memory)
                                                ^  ^
                                                ·  ·
  BºDATA QUERY (App. Monit)º                    ·  ·
                                                ·  ·
    Jaeger-UI    ←····→  jaeger-query ←·········┘  ·
    (Web React)          (GO lang)                 ·
     (Alt.1)                                       ·
                                                   ·
       Spark ······································┘
     (Alt.2) 

Disruptor
  Disruptor
  See also
  lmax-exchange.github.com

  When Disruptor is not good fit

GraphQL
• GraphQL: Facebook-developed language that provides a powerful API to get only
  the dataset you need in a single request, seamlessly combining data sources.

• REF: https://www.redhat.com/en/topics/api/what-is-graphql
  The purpose of this monorepo is to give the GraphQL Community:
      a to-specification official language service (see: API Docs)
      a comprehensive LSP server and CLI service for use with IDEs
      a codemirror mode
      a monaco mode (in the works)
      an example of how to use this ecosystem with GraphiQL.
      examples of how to implement or extend GraphiQL.
  
Apollo GraphQL Middelware
@[https://www.apollographql.com/]
· Simplify app development by combining APIs, databases, and microservices into a
  single data graph that you can query with GraphQL.
· @[https://www.infoq.com/news/2020/06/apollo-platform-graphql/]
  Apollo Data Graph Platform: A GraphQL Middleware Layer for the Enterprise

Ambari: Hadoop Cluster Provision
@[https://projects.apache.org/project.html?ambari]
Apache Ambari makes Hadoop cluster provisioning, managing, and monitoring dead simple.

Spark
@[http://spark.apache.org/]
- General cluster computing framework 
- initially designed around the concept of Resilient Distributed Datasets (RDDs).
  - RDDs enable data reuse by persisting intermediate results in memory 
    - Use case 1: fast computations for iterative algorithms.
      - especially beneficial for work flows like machine learning 
        (Bºsame operation may be applied over and over againº)
    - Use case 2: clean and/or transform data.
- Speed
  - Run workloads 100x faster (compared to Hadoop?).
  - Apache Spark achieves high performance for both batch
    and streaming data, using a state-of-the-art DAG scheduler,
    a query optimizer, and a physical execution engine.

- Ease of Use
  - Write applications quickly in Java, Scala, Python, R, and SQL.
  - Spark offers over 80 high-level operators that make it easy to
    build parallel apps. And you can use it interactively from the
    Scala, Python, R, and SQL shells.
    - Example PythonBºDataFrame APIº.
      |Bºdfº=ºspark.read.jsonº("logs.json") ← automatic schema inference
      |Bºdfº.where("age ˃ 21")
      |    .select("name.first").show()


- Generality
  - Combine SQL, streaming, and complex analytics.
  - Spark powers a stack of libraries including SQL and DataFrames,
    MLlib for machine learning, GraphX, and Spark Streaming.
    You can combine these libraries seamlessly in the same application.

- Runs Everywhere
  - Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone,
    or in the cloud. It can access diverse data sources.

- You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN,
  on Mesos, or on Kubernetes. Access data in HDFS, Alluxio, Apache Cassandra,
  Apache HBase, Apache Hive, and hundreds of other data sources.

- Common applications for Spark include real-time marketing campaigns, online
  product recommendations, cybersecurity analytics and machine log monitoring.


Apache Druid
- Complementary to Spark, Druid can be used to accelerate OLAP queries in Spark.
  - Spark: indicated for BºNON-interactiveº latencies.
  - Druid: indicated for Bº    interactiveº (sub-milisec) scenarios.
    - Use-case:
      - powering applications used by thousands of users where each query
        must return fast enough such that users can interactively explore 
        through data.
  - Druid fully indexes all data, and Bºcan act as a middle layer between º
  BºSpark and final appº.
BºTypical production setupº:
  
  data → Spark processing → Druid → Serve fast queries to clients

  More info at:
  https://www.linkedin.com/pulse/combining-druid-spark-interactive-flexible-analytics-scale-butani

Hadoop "vs" Spark
@[https://www.infoworld.com/article/3014440/big-data/five-things-you-need-to-know-about-hadoop-v-apache-spark.html]
Hadoop is essentially a distributed data infrastructure:
 -It distributes massive data collections across multiple nodes
  within a cluster of commodity servers
 -It also indexes and keeps track of that data, enabling
  big-data processing and analytics far more effectively
  than was possible previously.
Spark, on the other hand, is a data-processing tool that operates on those
distributed data collections; it doesn't do distributed storage.

You can use one without the other:
  - Hadoop includes not just a storage component, known as the
  Hadoop Distributed File System, but also a processing component called
  MapReduce, so you don't need Spark to get your processing done.
  - Conversely, you can also use Spark without Hadoop. Spark does not come with
  its own file management system, though, so it needs to be integrated with one
  - if not HDFS, then another cloud-based data platform. Spark was designed for
  Hadoop, however, so many agree they're better together.

Spark is generally a lot faster than MapReduce because of the way it processes
data. While MapReduce operates in steps, Spark operates on the whole data set
in one fell swoop:
   "The MapReduce workflow looks like this: read data from the cluster, perform
    an operation, write results to the cluster, read updated data from the
    cluster, perform next operation, write next results to the cluster, etc.,"
    explained Kirk Borne, principal data scientist at Booz Allen Hamilton.
    Spark, on the other hand, completes the full data analytics operations
    in-memory and in near real-time:
    "Read data from the cluster, perform all of the requisite analytic
    operations, write results to the cluster, done," Borne said.
Spark can be as much as 10 times faster than MapReduce for batch processing and
p to 100 times faster for in-memory analytics, he said.
  You may not need Spark's speed. MapReduce's processing style can be just fine
if your data operations and reporting requirements are mostly static and you
can wait for batch-mode processing. But if you need to do analytics on
streaming data, like from sensors on a factory floor, or have applications that
require multiple operations, you probably want to go with Spark.
 Most machine-learning algorithms, for example, require multiple operations.

Recovery: different, but still good.
Hadoop is naturally resilient to system faults or failures since data
are written to disk after every operation, but Spark has similar built-in
resiliency by virtue of the fact that its data objects are stored in something
called resilient distributed datasets distributed across the data cluster.
"These data objects can be stored in memory or on disks, and RDD provides full
recovery from faults or failures," Borne pointed out.

Hive
Data W.H. on top of Hadoop
@[https://projects.apache.org/project.html?hive]
- querying and managing utilities for large datasets
  residing in distributed storage built on top of Hadoop
- easy data extract/transform/load (ETL)
 * a mechanism to impose structure on a variety of data formats*
- access to files stored in Apache HDFS and other data storage
  systems such as Apache HBase (TM)

HiveMQ Broker: MQTT+Kafka for IoT
@[https://www.infoq.com/news/2019/04/hivemq-extension-kafka-mqtt/]

Orange
@[https://orange.biolab.si/screenshots/]
Features:
- Interactive Data Visualization
- Visual Programming
- Student's friendly.
- Add-ons
- Python Anaconda Friendly
  $ conda config --add channels conda-forge
  $ conda install orange3
  $ conda install -c defaults pyqt=5 qt
- Python pip  Friendly
  $ pip install orange3

observability: 
      loging
+ monitoring
+    tracing
Ex:ºFile Integrity Monitoring at scale: (RSA Conf)º
@[https://www.rsaconference.com/writable/presentations/file_upload/csv-r14-fim-and-system-call-auditing-at-scale-in-a-large-container-deployment.pdf]
Auditing log to gain insights at scale:

                         ┌─→ Pagerduty
            ┌─→ Grafana ─┼─→ Email
   Elastic  │            └─→ Slack
   Search  ─┼─→ Kibana
            │
            └─→ Pre─processing ─→ TensorFlow

Alt1:
  User   │ go-audit-                          User space
  land   │ container                             app
  ───────├─────  Netlink ───── Syscall iface ───────────
  Kernel │        socket           ^
         │          ^              |
                    └─  Kauditd ───┘

Loggin@Coinbase
@[https://www.infoq.com/news/2019/02/metrics-logging-coinbase]

Adidas Reference Architecture
Source @[https://adidas.github.io]

ºuser interfaceº

Working with the latest JavaScript technologies as well as building a stable
ecosystem is our goal. We are not only using one of many available frameworks,
we test them, we analyze our necessities and we choose what has the best
balance between productivity and developer satisfaction.

adidas is moving most of the applications from the desktop to the browser, and
we are improving every day to achieve it in the less time possible:
standardization, good practices, continuous deployment, automated tasks are
part of our daily work.

Our stack has changed a lot since the 2015. Back then we used code which is
better not to speak about. Now, we power our web applications with frameworks
like React and Vue and modern tooling. Our de facto tool for shipping these
applications is webpack.

We love how JavaScript is changing everyday and we encourage the developers to
use its latest features. TypeScript is also a good choice for developing
consistent, highly maintainable applications.

• ºAPIº
Apiary is our collaborative platform for designing, documenting and governing
API's. It helps us to adopt API first approach with templates and best
practices. Platform is strongly integrated with our API guidelines and helps us
to achieve consistency of all our API's. Apiary also speeds up and simplifies
the development and testing phase by providing API mock services and automation
of API testing.

Mashery is an API platform that delivers key API management capabilities needed
for successful digital transformation initiatives. Its full range of
capabilities includes API creation, packaging, testing, and management of your
API's and community of users. API security is provided via an embedded or
optional on-premise API gateway.

Runscope is our continuous monitoring platform for all our API's we are
exposing. It monitors uptime, performance and validates the data. It is tightly
integrated with Slack and Opsgenie in order to alert and create incident if
things are going wrong way. Using Runscope we can detect any issues with our
API's super fast and act accordingly to achieve highest possible SLA.

ºfast dataº
Apache Kafka is the core of our Fast Data Platform.
The more we work with it, the more versatile we perceived it was, covering more
use cases apart of event sourcing strategies.

It is designed to implement the pub-sub pattern in a very efficient way,
enabling Data Streaming cases, where multiple can easily subscribe to the same
stream of information. The fan-out is the pub-sub pattern is implemented in a
really efficient way, allowing to achieve high thought in the production and
the consumption part.

It also enables other capabilities like Data Extraction and Data Modeling, as
long as Stateful Event Processing. Together with Storm, Kafka Streams and the
Elastic stack, provides the perfect toolset to integrate our digital
applications in a modern self-service way.

ºacidº
Jenkins has become the de facto standard for continuous integration/continuous
delivery projects.
ACID is a 100% docker powered by Kubernetes that allows you to create a Jenkins
as a Service platform. ACID provides an easy way to generate an isolated
Jenkins environment per team while keeping a shared basic configuration and a
central management dashboard.
Updating and maintaining teams instances have never been so easy. Team freedom
provided out of the box, gitops disaster recovery capabilities and elastic
slaves allocation are the key pillars.

ºtestingº
SerenityBDD is our default for automation in test solution. Built on top of
Selenium and Cucumber, it enables us to write cleaner and more maintainable
automated acceptance and regression tests in a faster way. With archetypes
available for both frontend (web) and backend (API), it is almost
straightforward to be implemented in most projects.

Supporting a wide type of protocols and application types, Neoload is our tool
of choice for performance testing. It provides an easy to use interface with
powerful recording capabilities, allowing manual implementation as well if the
user chooses. Since it supports dockerized load generators, it makes it easy to
create a shared infrastructure and integrate it in our continuous integration
chain.

ºmonitoringº
From the monitoring perspective, we are using different tools trying to provide
internally end-to-end monitoring, for a better troubleshooting.

For system and alerting monitoring, we are using Prometheus, an open source
solution, easy for developers and completely integrated with Kubernetes. For
infrastructure monitoring we have Icinga with hundreds of custom scripts and
graphite as timeseries database. About APM, Instana is our solution, which
makes easy and fast to monitor the applications and identify bottlenecks. We
use Runscope to test and monitor our APIs with continuous schedule to give us
completely visibility into API problems.

Grafana is the visualization tool by default which is integrated with
Prometheus, Instana, Elasticsearch, Graphite, etc.

For logging, we are based on the ELK stack running on-premises and AWS
Kubernetes, with custom components for processing and alerting being able to
notify the problems to the teams.

As the goal of monitoring is the troubleshooting, all these tools have
integration with Opsgenie, where we have defined escalation policies for
alerting and notifications. The incidents can be communicated to the final
users via Statuspage.

ºmobileº
At mobile development, we run out of hybrid solutions. Native is where we live.

We work with the latest and most concise, expressive and safe languages
available at the moment for android and iOS mobile phones: Kotlin and Swift.

Our agile mobile development process requires big doses of automation to test,
build and distribute reliably and automatically every single change in the app.
We rely on tools such as Jenkins and Fastlane to automate those steps.

100% Serverless AWS arch
What a typical 100% Serverless Architecture looks like in AWS!
@[https://medium.com/serverless-transformation/what-a-typical-100-serverless-architecture-looks-like-in-aws-40f252cd0ecb]

Example Data Store Arch
- Mix data store model based on:
  - Document store:
    - MongoDB        : Store and manage objects directly as documents, keeping their structure intact.
                       Avoiding Object Relational Mapping in RDMS.
    - MongoDB-GridFS : storage system supports efficient querying and storage of binary files.
                       (videos streamed).
                       Avoid the need fo blob storage structures.
  - Graph database: (Hyper-relational ddbb)
    - Neo4j: Allows for complex queries for highly linked data, stored data as nodes and links.

  - Spring Data module repositories: used to "match" Java codebase with underlying data store.
    - Allows mapping java objects to MongoDB with minimal effort.  

Facebook
netconsole monit@scale
@[http://www.serverwatch.com/server-news/linuxcon-how-facebook-monitors-hundreds-of-thousands-of-servers-with-netconsole.html]
"""""" Facebook had a system in the past for monitoring
that used syslog-ng, but it was less than 60 percent reliable.
In contrast, Owens stated netconsole is highly scalable and can
handle enormous log volume with greater than 99.99 percent
reliability.  """""""

Badoo
20 billion Events/day
@[https://www.infoq.com/news/2019/08/badoo-20-billion-events-per-day/]

External Links
- Documentation:
@[http://kafka.apache.org/documentation/]

- Clients: (Java,C/C--,Python,Go,...)
@[https://cwiki.apache.org/confluence/display/KAFKA/Clients]

- Official JAVA JavaDoc API:
 @[http://kafka.apache.org/11/javadoc/overview-summary.html]

- Papers, ppts, ecosystem, system tools, ...:
@[https://cwiki.apache.org/confluence/display/KAFKA/Index]

- Kafka Improvement Proposals (KIP)
@[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals]

- Data pipelines with Kotlin+Kafka+(async)Akka
@[../JAVA/kotlin_map.html?query=data+pipelines+kafka]

- Cheatsheets:
@[https://lzone.de/cheat-sheet/Kafka]
@[https://ronnieroller.com/kafka/cheat-sheet]
@[https://abdulhadoop.wordpress.com/2017/11/06/kafka-command-cheat-sheet/]
@[https://github.com/Landoop/kafka-cheat-sheet]

Global Diagram
(one+ server in 1+ datacenters)

☞Bºbest practiceº: publishers must be unaware of underlying 
   LOG partitions and only specify a partition key:
   While LOG partitions are identifiable and can be sent 
   to directly, it is not recommended. Higher level 
   constructs are recommended.

┌────────┐              ┌──────────┐               ┌───────────┐
│PRODUCER│ 1 ←······→ 1 │TOPIC     │ 1 ←···→ 0...N │SUBSCRIBERS│
│        ··→ stream of··→(CATEGORY)│               │           │
└────────┘ OºRECORDsº   └──────────┘               └───────────┘
  ↑          ======       1                          1 ← Normally, 1←→1, but not necesarelly
  |        Oº·key       º ↑                          ↑       
  ·        Oº·value     º ·                          ·     
  ·        Oº·timeStamp º ·                          ·
  ·                       ·                          ·  GºCONSUMER GROUPº: cluster of consumers
  ·                       ·                          ↓    |  (vs single process consumer) º*1º
(optionally) producers    ·                          1    ↓
can wait for ACK.         ↓                          CONSUMER   |    CONSUMER       ←───┐
  ·                       1                          GROUP A    |    GROUP B            │
  ·   ┌───┬────────────────────┬──────────────────┐  ---------  | --------------------   
  ·   │LOG│                    │  ordered records │  Cli1 Cli2  | Cli3 Cli4 Cli5 Cli6   │
  |   ├───┘                    │  ─────────────── │   |    |    |  |    |    |    |      
  └···→┌Partitionº0ºreplica ┌1 │  0 1 2 3 4 5     │  ←┤···········←┘    ·    ·    ·     │
      │└Partitionº0ºreplica │2┐│  0 1 2 3 4 5     │   ·    ·    |       ·    ·    ·      
      │┌Partition 1 replica ├1││  0 1 2 3         │   ·    ·    |       ·    ·    ·     │
      │└Partition 1 replica │2┤│  0 1 2 3         │  ·+···←┤    |       ·    ·    ·      
      │┌Partitionº2ºreplica ├1││  ...             │   ·    |    |       ·    ·    ·     │
      │└Partitionº2ºreplica │2┤│                  │   |···←┘    |       ·    ·    ·      
      │┌Partition 3 replica ├1││ - Partitions are │  ←┴················←┘    ·    ·     │
      │└Partition 3 replica │2┤│ independent and  │             |            ·    ·      
      │┌Partitionº4ºreplica ├1││ grow at differn. │             |            ·    ·     │
      │└Partitionº4ºreplica │2┤│ rates.           │             |            ·    ·      
      │┌Partition 5 replica ├1││                  │             |            ·    ·     │
      │└Partition 5 replica │2┤│ - Records expire │  ←······················←┘    ·      
      │┌Partitionº6ºreplica └1││ and can not be   │             |                 ·     │
      │└Partitionº6ºreplica  2┘│ deleted manually │  ←···························←┘      
      └──↑───────────────────^─┴──────────────────┘ * num. of group instances           │
                      │      │    ←── offset ─→       must be <= # partitions            
                      │      └────────────┐   record.offset (sequential id)             │
┌─────────────────────┴─────────────────┐ │   uniquelly indentifies the record           
─ Partitions serve several purposes:      │   within the partition.                     │
  ─ allow scaling beyond a single server. │                                              
  ─ act as the ☞BºUNIT OF PARALLELISMº.   │  - aGºCONSUMER GROUPºis a view (state,  ┐   │
─ Partition.ºRETENTION POLICYº indicates  │    position, or offset) of full LOG.    │
  how much time records will be available │  - consumer groups enable different     ├───┘
  for compsumption before being discarded.│    apps to have a different view of the │
-Bºnumber of partitions in an event hubº  │    LOG, and to read independently at    │
 Bºdirectly relates to the number of      │    their own pace and with their own    ┘
 Bºconcurrent readers expected.º          │    offsets:
-OºTOTAL ORDER  of events is just guaran-º│  ☞Bºin a stream processing Arch,º
 Oºteed inside a partition.º              │   Bºeach downstream applicationº 
   Messages sent by a producer to a given │   Bºequates to a consumer groupº
   partition are guaran. to be appended   │  
   in the order they were sent.           │  
          ┌───────────────────────────────┘           
- Messages sent by a producer to a particular topic partition are guaranteed
  to be appended in the order they are sent.
  ┌───────┴────────┐
  PARTITION REPLICA: (fault tolerance mechanism) 
  └ Kafka allows producers to wait on acknowledgement so that a write 
    isn't considered complete until it is fully replicated and guaranteed 
    to persist even if the server written to fails, allowing to balance 
    replica consistency vs performance.
  └ Each partition has one replica server acting as leader" and 0+ 
    replica servers "followers". The leader handles all read 
    and write requests for the partition while the followers passively 
    replicate the leader. If the leader fails, one of the followers 
    replaces it.
  └ A server can be a leader for partition A and a follower for 
    partition B, providing better load balancing.
  └ For a topic with replication factor N, we will tolerate up to N-1 
    server failures without losing any records committed to the log.

º*1:º
 - Log partitions are (dnyamically) divided over consumer instances so 
   that each client instance is the exclusive consumer of a "fair share"
   of partitions at any point in time.
 - The consumer group generalizes the queue and publish-subscribe:
   - As with a queue the consumer group allows you to divide (scale) up
     processing over a collection of processes (the members of the consumer group).
   - As with publish-subscribe, Kafka allows you to broadcast messages to
     multiple consumer groups

Apache AVRO format
• Efficient compac way to store data in disk.
• "Imported" from Hadoop's World. 
• Efficient: Binary format, containing only data (vs JSON/JSON-schema field name+data)
  libraries (avro-tools.jar, kafka-avro-console-consumer,...) used to write/read.
• Schema never goes inside the message, only the "SchemaID". 
• Support for (fast) Google’s Snappy Compression.
• Support forºon-demand deserialization of columns (vs full data)º.
• It's not a column store (like Parquet, supporting better compression and insertion    [comparative]
  predicates), but it adapts better to big fragments of data.
@[https://www.slideshare.net/HadoopSummit/file-format-benchmark-avro-json-orc-parquet]

• Example:
  JAVA Class         ←··→ Avro Schema 
  =================       ======================
  public class User {     {
      long   id;            "type": "record",
      String name;          "name": "User",
  }                         "fields" : [
                              {"name": "id"  , "type": "long"  },
                              {"name": "name", "type": "string"}
                            ]
                          }

• No data is null by default                             [million_dolar_mistake]

• Enterprise Patterns:
  • Define global or by-product dictionary of Business events.
  • Manage Avro schemas within a Source Control Management ( "Git" )
    (Notice also Avro support for schema evolution)
    "Dump" schemas and topic relations to Confluent Schema Registry 
    when new events are validated (or existing events updated).
    event validation: Get sure they are really needed 
    (they are NOT a duplicated view of other existing event).
  • Ideally each kafka topic corresponds to a single schema.
    A (very opinionated)  "HIERARCHICAL TOPIC NAMING" can be similar to:
    ${cluster}.${tenant}.${product_or_concern}.${recipient}.${version}
                                                └────┬────┘
                                              process, person or group in charge 
                                              of reacting to event.
    This hierarchy also help in classifying different topics into 
    different "Service Levels" categories (scalability, partitions, replication 
    factor, retention time, ...) 
 
@[https://shravan-kuchkula.github.io/kafka-schemas/#understand-why-data-schemas-are-a-critical-part-of-a-real-world-stream-processing-application]

Kafka Schema Registry
BºDZone Intro Summaryº
@[https://dzone.com/articles/kafka-avro-serialization-and-the-schema-registry]
by Jean-Paul Azar

Confluent Schema Registry:
  - REST API for producers/consumers managing Avro Schemas:
    - store schemas for keys and values of Kafka records.
    - List schemas and schema-versions by subject.
    - Return schema by version or ID.
    - get the latest version of a schema.
    - Check if a given schema is compatible with a certain version. 
       - Compatibility level include:
         - backward: data written with old schema readable with new one.
         - forward : data written with new schema is readable with old one
         - full    : backward + forward
         - none    : Schema is stored by not schema validation is disabled
                   (Rºnot recommendedº).
       - configured Bºglobally or per subjectº.

  - Compatibility settings can be set to support Bºevolution of schemasº.

  - Kafka Avro serialization project provides serializers 
    taking schemas as input?
  - Producers send the schema (unique) ID and consumers fetch (and cache)
    the full schema from the Schema Registry.

  - Producer will create a new Avro record (schema ID, data). 
    Kafka Avro Serializer will register (and cache locally) the associated
    schema if needed, before serializing the record.

  - Schema Registry ensures that producer and consumer see compatible
    schemas and Bºautomatically "transform" between compatible schemas,º
  Bºtransforming payload via Avro Schema Evolutionº.
   

BºSCHEMA EVOLUTIONº
  Scenario:
  - Avro schema modified after data has already been written to store
    with old schema version.
 
OºIMPORTANTº: ☞ From Kafka perspective,Oºschema evolutionº happens only
              Oºduring deserialization at the consumer (read)º:
                If consumer’s schema is different from the producer’s schema,
                and they are compatible, the value or key is automatically 
                modified during deserialization to conform to the consumer's
                read schema if possible.

BºSchema Evolution: Allowed compatible Modification º
- change/add field's default value.
- add new         field with default value.
- remove existing field with default value.
- change field's order attribute.
- add/remove      field-alias (RºWARN:º can break consumers depending on the alias).
- change type → union-containing-original-type.

BºBest Patternº
- Provide a default value for fields in your schema.
- Never change a field's data type.
- Do NOT rename an existing field (use aliases instead).

Ex: Original schema v1: 

{
  "namespace": "com.cloudurable.phonebook",
  "type": "record",
  "name": "Employee",
  "doc" : "...",
  "fields": [
    {"name": "firstName", "type": "string"                            },
    {"name": "nickName" , "type": ["null", "string"] , "default" : null},
    {"name": "lastName" , "type": "string"                             },
    {"name": "age"      , "type": "int"              , "default": -1   },
    {"name": "emails"   , "type": {"type" : "array",
                                   "items": "string"}, "default":[]    },
    {"name": "phoneNum" , "type":
                                  [ "null",
                                    { "type": "record",
                                      "name": "PhoneNum",
                                      "fields": [
                                        {"name": "areaCode"   , "type": "string"},
                                        {"name": "countryCode", "type": "string", "default" : ""},
                                        {"name": "prefix"     , "type": "string"},
                                        {"name": "number"     , "type": "string"}
                                      ]
                                    }
                                  ]
    },
    {"name": "status"                                , "default" :"SALARY", 

                        , "type": {
                                    "type": "enum",
                                    "name": "Status",
                                    "symbols" : ["RETIRED", "SALARY",...]
                                  }                  
    }
  ]
}

- Schema Version 2: 
  "age" field, def. value -1, added


               | KAFKA LOG |
  Producer@v2 →|Employee@v2| ··→ consumer@v.1 ·····→ NoSQL Store
               |           |     ^                    ^
               |           |     age field removed    'age' missing
               |           |     @deserialization
               |           |     
               |           |     
               |           |     consumer@ver.2 ←..... NoSQL Store
               |           |     ^                     ^
               |           |     age set to -1         'age' missing


BºRegistry REST API Ussageº

  └ POST New Schema
$º$ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \º
$º$   --data '{"schema": "{\"type\": …}’ \                                  º
$º$   http://localhost:8081/subjects/Employee/versions                      º

  └ List all of the schemas:

$º$ curl -X GET http://localhost:8081/subjects                              º

    Using Java OkHttp client:

    package com.cloudurable.kafka.schema;
    import okhttp3.*;
    import java.io.IOException;
    public class SchemaMain {
        private final static 
        MediaType SCHEMA_CONTENT =
                MediaType.parse("application/vnd.schemaregistry.v1+json");
        private final static String EMPLOYEE_SCHEMA = "{ \"schema\": \"" ...;
        private final static String BASE_URL = "http://localhost:8081";
     
    
        private final OkHttpClient client = new OkHttpClient();

        private static newCall(Requeste request) {
            System.out.println(client.newCall(request).execute().body().string() );
        }
        private static putAndDumpBody (final String URL, RequestBody BODY) {
            newCall(new Request.Builder().put(BODY).url(URL).build());
        }
        private static postAndDumpBody(final String URL, RequestBody BODY) {
            newCall(new Request.Builder().post(BODY).url(URL).build());
        }
    
        private static getAndDumpBody(final String URL) {
            request = new Request.Builder() 
                    .url(URL).build();
            System.out.println(client.newCall(request).
                   execute().body().string());
        }
    
        public static void main(String... args) throws IOException {
            System.out.println(EMPLOYEE_SCHEMA);
    
            postAndDumpBody(                            // ← POST A NEW SCHEMA
              BASE_URL + "/subjects/Employee/versions",
              RequestBody.create( SCHEMA_CONTENT, EMPLOYEE_SCHEMA )
            );
    
            getAndDumpBody(BASE_URL + "/subjects");     // ← LIST ALL SCHEMAS
    
            getAndDumpBody(BASE_URL                     // ← SHOW ALL VERSIONS
                 + "/subjects/Employee/versions/");
    
            getAndDumpBody(BASE_URL                     // ← SHOW VERSION 2 OF EMPLOYEE
                 + "/subjects/Employee/versions/2");
    
            getAndDumpBody(BASE_URL                     // ← "SHOW SCHEMA WITH ID 3
                 + "/schemas/ids/3");
    
            getAndDumpBody(BASE_URL                     // ← SHOW LATEST VERSION
                 + "/subjects/Employee/versions/latest");
    
            postAndDumpBody(                            // ← SCHEMA IS REGISTERED?
              BASE_URL + "/subjects/Employee",
              RequestBody.create( SCHEMA_CONTENT, EMPLOYEE_SCHEMA )
            );
    
            postAndDumpBody(                            // ← //TEST COMPATIBILITY
              BASE_URL + "/compatibility/subjects/Employee/versions/latest",
              RequestBody.create( SCHEMA_CONTENT, EMPLOYEE_SCHEMA )
            );
    
            getAndDumpBody(BASE_URL                     // ← TOP LEVEL CONFIG
                 + "/config");
    
            putAndDumpBody(                            // ← SET TOP LEVEL CONFIG VALs
              BASE_URL + "/config",                    //   VALs :=none|backward|
              RequestBody.create(SCHEMA_CONTENT,                   forward|full
                 "{\"compatibility\": \"none\"}"
            );
    
            putAndDumpBody(                            // ← SET CONFIG FOR EMPLOYEE
              BASE_URL + "/config/Employee",          //
              RequestBody.create(SCHEMA_CONTENT,
                 "{\"compatibility\": \"backward\"}"
            );
        }
    }

BºRUNNING SCHEMA REGISTRYº
  $º$ CONFIG="etc/schema-registry/schema-registry.properties"º
  $º$ cat ${CONFIG}                                          º
  $ºlisteners=http://0.0.0.0:8081                            º
  $ºkafkastore.connection.url=localhost:2181                 º
  $ºkafkastore.topic=_schemas                                º
  $ºdebug=false                                              º

  $º$ .../bin/schema-registry-start ${CONFIG}                º
  
BºWriting Producers/Consumers with Avro Serializers/Sche.Regº

  └ start up the Sch.Reg. pointing to ZooKeeper(cluster).
  └ Configure gradle: 
    plugins {
      id "com.commercehub.gradle.plugin.avro" version "0.9.0"
    }     └─────────────┬──────────────────┘
    //    http://cloudurable.com/blog/avro/index.html
    //    transform Avro type → Java class
    //    Plugin supports:
    //    - Avro schema files (.avsc) ("Kafka")
    //    - Avro RPC IDL (.avdl)
    // $º$ gradle buildº ← generate java classesº

    group 'cloudurable'
    version '1.0-SNAPSHOT'
    apply plugin: 'java'
    sourceCompatibility = 1.8
    dependencies {
      testCompile 'junit:junit:4.11'
      compile 'org.apache.kafka:kafka-clients:0.10.2.0'  ← 
      compile "org.apache.avro:avro:1.8.1"               ← Avro lib
      compile 'io.confluent:kafka-avro-serializer:3.2.1' ← Avro Serializer
      compile 'com.squareup.okhttp3:okhttp:3.7.0'
    }
    repositories {
        jcenter()
        mavenCentral()
        maven { url "http://packages.confluent.io/maven/" }
    }
    avro {
        createSetters = false
        fieldVisibility = "PRIVATE"
    }
  └ Setup producer to use GºSchema Registryº and BºKafkaAvroSerializerº
    package com.cloudurable.kafka.schema;
    import com.cloudurable.phonebook.Employee;
    import com.cloudurable.phonebook.PhoneNum;
    import io.confluent.kafka.serializers.KafkaAvroSerializerConfig;
    import org.apache.kafka.clients.producer.KafkaProducer;
    import org.apache.kafka.clients.producer.Producer;
    import org.apache.kafka.clients.producer.ProducerConfig;
    import org.apache.kafka.clients.producer.ProducerRecord;
    import org.apache.kafka.common.serialization.LongSerializer;
    import io.confluent.kafka.serializers.KafkaAvroSerializer;
    import java.util.Properties;
    import java.util.stream.IntStream;
    
    public class AvroProducer {
        private static Producer˂Long, Employee˃ createProducer() {
            final String
              serClassName = LongSerializer.class.getName();
              KafkaAvroClN = Serializer    .class.getName();
              SCHEMA_REG_URL_CONFIG = KafkaAvroSerializerConfig.
                                      SCHEMA_REGISTRY_URL_CONFIG;
              VAL_SERI_CLASS_CONFIG = ProducerConfig.
                                      VALUE_SERIALIZER_CLASS_CONFIG
    
            final Properties props = new Properties();
            props.put(ProducerConfig.   BOOTSTRAP_SERVERS_CONFIG , "localhost:9092");
            props.put(ProducerConfig.           CLIENT_ID_CONFIG , "AvroProducer"  );
            props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG , serClassName    );
          Bºprops.put(                     VAL_SERI_CLASS_CONFIG , KafkaAvroClN    );º
          Bº//        └────────────────────────┬──────────────────────────────┘      º
          Bº//        CONFIGURE KafkaAvroSerializer.                                 º
          Gºprops.put(                     SCHEMA_REG_URL_CONFIG , º // ← Set Schema Reg.
          Gº                              "http://localhost:8081");º      URL
            return new KafkaProducer˂˃(props);
        }
    
        private final static String TOPIC = "new-employees";
    
        public static void main(String... args) {
            Producer˂Long, Employee˃ producer = createProducer();
            Employee bob = Employee.newBuilder().setAge(35)
                    .setFirstName("Bob").set...().build();
            IntStream.range(1, 100).forEach(index->{
                producer.send(new ProducerRecord˂˃(TOPIC, 1L * index, bob));
            });
            producer.flush();
            producer.close();
        }
    }
  └ Setup consumer to use GºSchema Registryº and BºKafkaAvroSerializerº
    
    package com.cloudurable.kafka.schema;
    import com.cloudurable.phonebook.Employee;
    import io.confluent.kafka.serializers.KafkaAvroDeserializer;
    import io.confluent.kafka.serializers.KafkaAvroDeserializerConfig;
    import org.apache.kafka.clients.consumer.Consumer;
    import org.apache.kafka.clients.consumer.ConsumerConfig;
    import org.apache.kafka.clients.consumer.ConsumerRecords;
    import org.apache.kafka.clients.consumer.KafkaConsumer;
    import org.apache.kafka.common.serialization.LongDeserializer;
    import java.util.Collections;
    import java.util.Properties;
    import java.util.stream.IntStream;
    public class AvroConsumer { 
      private final static String BOOTSTRAP_SERVERS = "localhost:9092";
      private final static String TOPIC = "new-employees";
      private static Consumer createConsumer() {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "KafkaExampleAvroConsumer");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
                  LongDeserializer.class.getName());
        //USE Kafka Avro Deserializer.
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
          KafkaAvroDeserializer.class.getName());
    
        //Use Specific Record or else you get Avro GenericRecord.
        props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true");
    
        //Schema registry location.
        props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG,
                "http://localhost:8081"); //<----- Run Schema Registry on 8081
        return new KafkaConsumer<>(props);
      }

      public static void main(String... args) {
          final Consumer consumer = createConsumer();
          consumer.subscribe(Collections.singletonList(TOPIC));
          IntStream.range(1, 100).forEach(index -> {
              final ConsumerRecords records =
                      consumer.poll(100);
              if (records.count() == 0) {
                  System.out.println("None found");
              } else records.forEach(record -> {
                  Employee employeeRecord = record.value();
                  System.out.printf("%s %d %d %s \n", record.topic(),
                          record.partition(), record.offset(), employeeRecord);
              });
          });
      }
    }

    Notice that just like with the producer, we have to tell the 
    consumer where to find the Registry, and we have to configure the 
    Kafka Avro Deserializer.

Configuring Schema Registry for the consumer:

//Use Kafka Avro Deserializer.

props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,

                KafkaAvroDeserializer.class.getName());  

//Use Specific Record or else you get Avro GenericRecord.

props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true"); //Schema registry location.
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG,
                "http://localhost:8081"); //<----- Run Schema Registry on 8081

An additional step is that we have to tell it to use the generated 
version of the Employee object. If we did not, then it would use the 
Avro GenericRecord instead of our generated Employee object, which is 
a SpecificRecord. To learn more about using GenericRecord and 
generating code from Avro, read the Avro Kafka tutorial as it has 
examples of both.


https://docs.confluent.io/current/schema-registry/index.html
https://docs.confluent.io/current/schema-registry/schema_registry_tutorial.html

Pixy
@[https://github.com/mailgun/kafka-pixy]
  Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with 
  automatic consumer group control. It is designed to hide the 
  complexity of the Kafka client protocol and provide a stupid simple 
  API that is trivial to implement in any language.

Docker Img For Developers
@[https://github.com/lensesio/fast-data-dev]
- Apache Kafka docker image for developers; with Lenses (lensesio/box) 
  or Lenses.io's open source UI tools (lensesio/fast-data-dev). Have a 
  full fledged Kafka installation up and running in seconds and top it 
  off with a modern streaming platform (only for kafka-lenses-dev), 
  intuitive UIs and extra goodies. Also includes Kafka Connect, Schema 
  Registry, Lenses.io's Stream Reactor 25+ Connectors and more.

  Ex SystemD integration file:
  /etc/systemd/system/kafkaDev.service
  | #!/bin/bash
  | 
  | # Visit http://localhost:3030 to get into the fast-data-dev environment
  | 
  | [Unit]
  | Description=Kafka Lensesio
  | After=docker.service
  | Wants=network-online.target docker.socket
  | Requires=docker.socket
  | 
  | [Service]
  | Restart=always
  | # Create container if it doesn't exists with container inspect
  | ExecStartPre=/bin/bash -c "/usr/bin/docker container inspect lensesio/fast-data-dev 2> /dev/null || /usr/bin/docker run -d --name kafkaDev --net=host -v /var/backups/DOCKER_VOLUMES_HOME/kafkaDev:/data lensesio/fast-data-dev" 
  | ExecStart=/usr/bin/docker start -a    kafkaDev 
  | ExecStop=/usr/bin/docker   stop -t 20 kafkaDev 
  | 
  | [Install]
  | WantedBy=multi-user.target

Quick start
@[http://kafka.apache.org/documentation.html#quickstart]
BºPRE-SETUPº
  Download tarball from mirror @[https://kafka.apache.org/downloads],
  then untar like:
  $º$ tar -xzf - kafka_2.13-2.5.0.tgz ; cd kafka_2.13-2.5.0º

  - Start the server
  $º$ ~ bin/zookeeper-server-start.sh \                    º ← Start Zookeeper
  $º  config/zookeeper.properties 1˃zookeeper.log 2˃⅋1 ⅋   º

  $º$ ~ bin/kafka-server-start.sh  \                       º ← Start Kafka Server
  $º  config/server.properties 1˃kafka.log  2˃⅋1 ⅋ \       º

  $º$ bin/kafka-topics.sh --create --zookeeper \           º ← Create BºTESTº topic
  $º  localhost:2181  --replication-factor 1 \             º (Alt.brokers can be
  $º  --partitions 1 --topic TEST                          º  configured to  auto-create 
                              └──┘                            them when publishing to
                                                              non-existent ones)

  $º$ bin/kafka-topics.sh --list --zookeeper localhost:2181º ← Check topic
  $º$ TEST                                                 º ← Expected output           
      └──┘

  $º$ bin/kafka-console-producer.sh --broker-list          º ← Send some messages
  $º  localhost:9092 --topic TEST                          º
  $ºThis is a message                                      º
  $ºThis is another message                                º
  $ºCtrl+V                                                 º

  $º$ bin/kafka-console-consumer.sh --bootstrap-server \   º ← Start a consumer
  $ºlocalhost:9092 --topic Bºtestº --from-beginning        º
  $ºThis is a message                                      º
  $ºThis is another message                                º


  ********************************
  * CREATING A 3 BROKERS CLUSTER *
  ********************************
  $º$ cp config/server.properties config/server-1.propertiesº ← Add 2 new broker configs.
  $º$ cp config/server.properties config/server-2.propertiesº
                                  └───────────┬────────────┘ 
               ┌──────────────────────────────┤
   ┌───────────┴────────────┐      ┌──────────┴─────────────┐  
  ºconfig/server-1.properties:º   ºconfig/server-2.properties:º
   broker.id=1                     broker.id=2                 ← unique id 
   listeners=PLAINTEXT://:9093     listeners=PLAINTEXT://:9094
   log.dir=/tmp/kafka-logs-1       log.dir=/tmp/kafka-logs-2   ← avoid overwrite


  $º$ bin/kafka-server-start.sh config/server-1.properties ...º ← Start 2nd cluser
  $º$ bin/kafka-server-start.sh config/server-2.properties ...º ← Start 3rd cluser

  $º$ bin/kafka-topics.sh --create --zookeeper localhost:2181\º ← Create new topic with
  $º  --replication-factor 3 --partitions 1                   º Bºreplication factor of 3º
  $º  --topic  topic02                                        º

  $º$ bin/kafka-topics.sh --describe \                        º ← Check know which broker
  $º  --zookeeper localhost:2181 --topic topic02              º   is doing what
   (output will be similar to)
   Topic: topic02  PartitionCount:1  ReplicationFactor:3 Configs:         ← summary 
       Topic: topic02 Partition:0 Leader:1  Replicas: 1,2,0 Isr: 1,2,0 ← Partition 0
                                                            └┬┘
                                            set of "in-sync" replicas. 
                                           (subset of "replicas" currently
                                            alive and in sync with "leader")


  *****************************************
  * Using Kafka Connect to import/export  *
  * data using simple connectors          *
  *****************************************

BºPRE-SETUPº
  $º$ echo -e "foo\nbar" > test.txt       º ← Prepare (input)test data

  - SETUP source/sink Connectors:
    config/connect-file-srcs.properties ← unique_connector_id, connector class,
    config/connect-file-sink.properties   ...

  $º$ bin/connect-standalone.sh \            º ← Start Bºtwo connectorsº running in 
  $º  \                                      º   standalone mode (dedicated process)
  $º  config/connect-standalone.properties \ º ← 1st param is common Kafka-Connect config 
  $º  \                                      º   (brokers, serialization format ,...)
  $º  config/connect-file-srcs.properties  \ º 
  $º  config/connect-file-sink.properties    º 
    └─────────────────┬────────────────────┘
    examples in kafka distribution set the "pipeline" like:
    "test.txt" → connect-test  → sink connector → "test.sink.txt"


• See also: Ansible Install:
@[https://github.com/confluentinc/cp-ansible]
@[https://docs.confluent.io/current/installation/cp-ansible/index.html]
  - Installs Confluent Platform packages.
    - ZooKeeper
    - Kafka
    - Schema Registry
    - REST Proxy
    - Confluent Control Center
    - Kafka Connect (distributed mode)
    - KSQL Server
  
  - systemd Integration
  
  - Configuration options for:
     plaintext, SSL, SASL_SSL, and Kerberos.


Configuration
• Broker Config:
@[http://kafka.apache.org/documentation/#configuration]
  Main params:
  - broker.id
  - log.dirs
  - zookeeper.connect

• Topics config: @[http://kafka.apache.org/documentation/#topicconfigs"

• Compaction: kafka.devops, kakfa.performance
http://kafka.apache.org/documentation.html#compaction

Strimzi k8s Operator
  "Kafka on K8s in a few minutes"
• Strimzi: RedHat OOSS project that provides container images and 
  operators for running production-ready Kafka on k8s and OpenShift. 
• K8s native experience, using kubectl to manage the Kafka cluster and GitOps.
• Monitoring and observability integration with Prometheus.
• TODO:
@[https://developers.redhat.com/blog/2019/06/06/accessing-apache-kafka-in-strimzi-part-1-introduction/]
@[https://developers.redhat.com/blog/2019/06/07/accessing-apache-kafka-in-strimzi-part-2-node-ports/]
@[https://developers.redhat.com/blog/2019/06/10/accessing-apache-kafka-in-strimzi-part-3-red-hat-openshift-routes/]
@[https://developers.redhat.com/blog/2019/06/11/accessing-apache-kafka-in-strimzi-part-4-load-balancers/]
@[https://developers.redhat.com/blog/2019/06/12/accessing-apache-kafka-in-strimzi-part-5-ingress/]

 

Pulsar vs Kafka
- Pulsar is a younger project Bºinspired and informed by Kafkaº. 
  Kafka on the other side has a bigger community. 

- Pulsar Webinars                                          [resource]
@[https://streamnative.io/resource#pulsar]


BºPulsas "PRO"s over Kafkaº
Bº========================º
@[https://kafkaesque.io/7-reasons-we-choose-apache-pulsar-over-apache-kafka/]
@[https://kesque.com/5-more-reasons-to-choose-apache-pulsar-over-kafka/]

1. Streaming and queuing Come together:
   Pulsar supports standard message queuing patterns, such as
   competing consumers, fail-over subscriptions, and easy message 
   fan out keeping track of the client read position in the topic 
   and stores that information in its high-performance distributed 
   ledger, Apache BookKeeper, handling many of the use cases of a 
   traditional queuing system, like RabbitMQ.
 
2. Simpler ussage:
   If you don't need partition you don't have to worry about them.
   "If you just need a topic, then use a topic". Do not worry about
   how many consumers the topic might have. 
   Pulsar subscriptions allow you to add as many consumers as you want 
   on a topic with Pulsar keeping track of it all. If your consuming 
   application can’t keep up, you just use a shared subscription to 
   distribute the load between multiple consumers.
   Pulsar hasºpartitioned topicsºif you need them, but 
  ºonly if you need themº.

3. Fitting a log on a single server becomes a challenge (Disk full,
   remote copy o large logs can take a long time.
   More info at "Adding a New Broker Results in Terrible Performance"
 @[https://www.confluent.io/blog/stories-front-lessons-learned-supporting-apache-kafka/]
   Apache Pulsar breaks logs into segments and distributes them across
   multiple servers while the data is being written by using BookKeeper
   as its storage layer. 
   This means that the Bºlog is never stored on a single serverº, so a
   single server is never a bottleneck. Failure scenarios are easier to
   deal with and Bºscaling out is a snap: Just add another server.º
 BºNo rebalancing needed.º

4. Stateless Brokers:
   In Kafka each broker contains the complete log for each of
   its partitions. If load gets too high, you can't simply add
   another broker. Brokers must synchronize state from other 
   brokers that contain replicas of its partitions.
   In Pulsar brokers accept data from producers and send data
   to consumers, but the data is stored in Apache BookKeeper.
   If load gets high, just add another broker.
   It starts up quickly and gets to work right away.

5. Geo-replication is a first-class feature in Pulsar.
   (vs proprietary add-on).
 BºConfiguring it is easy and it just works. No PhD needed.º
   
6. Consistently Faster
 BºPulsar delivers higher throughput along with lower and moreº
 Bºconsistent latency.º

7. All Apache Open Source
   input and output connectors (Pulsar IO), SQL-based topic queries 
   (Pulsar SQL), schema registry,...
   (vs Kafka open-source features controlled by a commercial entity)

8. Pulsar can have multiple tenants and those tenants can have 
   multiple namespaces to keep things all organized. Add to that
   access controls, quotas, and rate-limiting for each namespace 
   and you can imagine a future where we can all get along using
   just this one cluster.
   (WiP or Kafka in KIP-37).

9. Replication
   You want to make sure your messages never get lost. In Kakfa
   you configure 2 or 3 replicas of each message in case 
   something goes wrong.
   In Kafka the leader stores the message and the followers make
   a copy of it. Once enough followers acknowledge they’ve got it,
   "Kafka is happy".
   Pulsar uses a quorum model: It sends the message out to a 
   bunch of nodes, and once enough of them acknowledge they've
   got it, "Pulsar is happy". Majority always wins, and all votes 
   are equal giving more consistent latency behavior over time.
 @[https://kafkaesque.io/performance-comparison-between-apache-pulsar-and-kafka-latency/]
   (Kafka quorum is also a WiP in KIP-250)

10.Tiered storage.
   What if you like to store messages forever (event-sourcing)?
   It can get expensive on main high-performance SSDs.
 BºWith Pulsar tiered storage you can automatically push old messages   º
 Bºinto practically infinite, cheap cloud storage (S3) and retrieve themº
 Bºjust like you do those newer, fresh-as-a-daisy messages.º
   (Kafka describes this feature in KIP-405).

11.End-to-end encryption
   Producer Java client can encrypt message using shared keys with
   the consumer. (hidden info to broker).
   Kafka is in feature-request state (KIP-317).

12.When an (stateless) Pulsar broker gets overloaded, pulsar rebalance
   request from clients automatically.
   It monitors the broker ussage of CPU, memory, and network (vs disk, since 
   brokers is stateless) to take the decision to balance.
   No need to add a new broker until all brokers are at full.
   In Kafka load-balancing is done by  installing another package 
   such as LinkedIn's Cruise Control or paying for Confluent's rebalancer
   tool.

— See also: [TODO]
@[https://streamnative.io/blog/tech/pulsar-vs-kafka-part-1]

KubeMQ: Kafka alternative
@[https://dzone.com/articles/seamless-migration-from-kafka-to-kubemq?edition=699391]
• Non free product.
  · Free for stand alone single pod.
  · Professional for SMBs: Deploy on up to 20 k8s clusters.
  · Professional for Dist. product: Corporate license .
  · Enterprise: Enhanced SLA spport.
• Kafka monolithic architecture is more suited for on-premise clusters or high-end multi-VM setups. Spinning up a multi-node cluster on a standalone workstation for testing purposes can be a challenge. 
• KubeMQ messaging service is built from the ground up with Kubernetes in mind:
  - It is Statless  and Ephemeral.
  - When config. changes are needed, nodes are shut down and replaced requiring 
  - Container images is "just" 30MB, suitable for local development.
   ºzeroº-configuration setup or post-install tweakingº:
    A 'channel' is the only object developers need to create (forget about 
    brokers, exchanges, orchestrators ... KubeMQ's Raft replaces ZooKeeper).
• KubeMQ message delivery patterns include:
  • Pub/Sub with or without persistence
  • Request/Reply (synchronous, asynchronous)
  • At Most Once Delivery
  • At Least Once Delivery
  • Streaming patterns
  • RPC
  (Comparatively, Kafka only supports Pub/Sub with persistence and streaming. 
   RPC and Request/Reply patterns are not supported by Kafka at all).

• kubemqctl: cli interface analogous to kubectl.
• KubeMQ Sources allows to connect to an existing Kafka input topic to simplify 
  migration from Kafka.

Akka+Kotlin:Data Pipes
@[https://www.kotlindevelopment.com/data-pipelines-kotlin-akka-kafka/]
- Objective: 
  Create a "GitHub monitor" that build an analytics component that
  polls one of your services (GitHub), writes data into a message queue
  for later analysis, then, after post-processing, updates statistics in
  a SQL database.

- Tooling: Akka Streams + Alpakka connector collection

- "Architecture":
  1) polls GitHub Event API for kotlin activity
  2) writes all events into a Kafka topic for later use
  3) reads events from Kafka and filters out PushEvents
  4) updates a Postgres database with:
     - who pushed changes
     - when
     - into which repository.

- Akka summary: data is moving from Sources to Sinks.
  (Observable and Sink in RxJava)

ºENTRY POINTº
(standard main function)
  fun main(vararg args: String) {
    val system = ActorSystem.create()                          // "boilerplate" for using Akka and Akka Streams
    val materializer = ActorMaterializer.create(system)
    val gitHubClient = GitHubClient(system, materializer)      // instance used to poll the GitHub events API
    val eventsProducer = EventsProducer(system, materializer)  // instance used to write events into Kafka
    val eventsConsumer = EventsConsumer(system)                // instance used to read events from Kafka
    val pushEventProcessor = PushEventProcessor(materializer)  // instance used to filter PushEvents and update the database
                                                               
    eventsProducer.write(gitHubClient.events())                // put things in motion.
    pushEventProcessor.run(eventsConsumer.read())
  }

  Each time we receive a response from GitHub, we parse it and send individual events downstream.
  fun events(): Source˂JsonNode, NotUsed˃ =
    poll().flatMapConcat { response -˃
      response.nodesOpt
        .map { nodes -˃ Source.from(nodes) }
        .orElse(Source.empty())
    }

ºEventsProducer and EventsConsumerº
(the "power" of Akka Streams and Alpakka)

- Akka-Streams-Kafka greatly reduces the amount of code
  that we have to write for integrating with Kafka.
  Publishing events into a Kafka topic look like:

  fun write(events: Source˂JsonNode, NotUsed˃)
      : CompletionStage˂Done˃
        =  events.map {                                        // ← maps GitHub event to(Kafka)ProducerRecord 
             node -˃ ProducerRecord˂ByteArray, String˃
                        ("kotlin-events",
                         objectMapper.writeValueAsString(node) // ← serialize JsonNode as a String
                        ) 
      }.runWith(                        // ← connects Source Sink  
          Producer.plainSink(settings), materializer)                                         
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        - defined@Akka Streams Kafka
        - takes care of communicating with Kafka

- The other way around, reading from Kafka, is also super simple.

  fun read(): Source˂JsonNode, NotUsed˃ =
    Consumer.plainSource(
        settings,
        Subscriptions.assignmentWithOffset(
            TopicPartition("kotlin-events", 0), 0L))
       .map { record -˃ objectMapper.readTree(record.value()) }
       .mapMaterializedValue { NotUsed.getInstance() }

  At this point, we have a copy of GitHub's events feed for github.com/Kotlin stored in Kafka,
so we can time travel and run different analytics jobs on our local dataset.

ºPushEventProcessorº
 we want to filter out PushEvents from the stream
 and update a Postgres database with the results.
 - Alpakka Slick (JDBC) Connector is used to connect to PostgreSQL.
   fun createTableIfNotExists(): Source˂Int, NotUsed˃ {
     val ddl =
       """
         |CREATE TABLE IF NOT EXISTS kotlin_push_events(
         |  id         BIGINT    NOT NULL,
         |  name       VARCHAR   NOT NULL,
         |  timestamp  TIMESTAMP NOT NULL,
         |  repository VARCHAR   NOT NULL,
         |  branch     VARCHAR   NOT NULL,
         |  commits    INTEGER   NOT NULL
         |);
         |CREATE UNIQUE INDEX IF NOT EXISTS id_index ON kotlin_push_events (id);
       """.trimMargin()
     return Slick.source(session, ddl, { _ -˃ 0 })
   }

 - Similarly, the function to update the database looks like this.

   fun Source˂PushEvent, NotUsed˃.updateDatabase() :
       CompletionStage˂Done˃ =
           createTableIfNotExists().flatMapConcat { this }
       .runWith(Slick.sink˂PushEvent˃(session, 20, { event -˃
         """
           |INSERT INTO kotlin_push_events
           |(id, name, timestamp, repository, branch, commits)
           |VALUES (
           |  ${event.id},
           |  '${event.actor.login}',
           |  '${Timestamp.valueOf(event.created_at)}',
           |  '${event.repo.name}',
           |  '${event.payload.ref}',
           |  ${event.payload.distinct_size}
           |)
           |ON CONFLICT DO NOTHING
         """.trimMargin()
       }), materializer)
 
   We are almost done, what's left is filtering and mapping from JsonNode to
   PushEvent and composing the methods together.

   fun Source˂JsonNode, NotUsed˃.filterPushEvents(): Source˂PushEvent, NotUsed˃ =
     filter { node -˃ node["type"].asText() == "PushEvent" }
       .map { node -˃ objectMapper.convertValue(node, PushEvent::class.java) }

   And finally, all the functions composed together look like this.
   fun run(events: Source˂JsonNode, NotUsed˃): CompletionStage˂Done˃ =
     events
       .filterPushEvents()
       .updateDatabase()

  This is why we've used the extension methods above, so we can describe 
  transformations like this, simply chained together. 
  That's it, after running the app for a while (gradle app:run) we can see 
  the activities around different Kotlin repositories. 


You can find the complete source code on GitHub.

    A very nice property of using Akka Streams and Alpakka is that it makes 
  really easy to migrate/reuse your code, e.g. in case you want to store data 
  in Cassandra later on instead of Postgres. All you would have to do is define 
  a different Sink with CassandraSink.create. Or if GitHub events would be 
  dumped in a file located in AWS S3 instead of published to Kafka, all you 
  would have to do is create a Source with S3Client.download(bucket, key). The 
  current list of available connectors is located here, and the list is growing.

Unordered
• Security:
  https://kafka.apache.org/documentation/#security
• Burrow Monit:
@[https://dzone.com/articles/kafka-monitoring-with-burrow]
• Best Pracites:
@[https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment]

• Mirror Maker (geo-replica):
@[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330]
  fka's mirroring feature makes it possible to maintain a replica of an 
  existing Kafka cluster. The following diagram shows how to use the 
  MirrorMaker tool to mirror a source Kafka cluster into a target 
  (mirror) Kafka cluster. The tool uses a Kafka consumer to consume 
  messages from the source cluster, and re-publishes those messages to 
  the local (target) cluster using an embedded Kafka producer.

• Kafka Backup:
@[https://github.com/itadventurer/kafka-backup]
- Kafka Backup is a tool to back up and restore your Kafka data 
  including all (configurable) topic data and especially also consumer 
  group offsets. To the best of our knowledge, Kafka Backup is the only 
  viable solution to take a cold backup of your Kafka data and restore 
  it correctly.

• Leasson Learned:
@[https://www.confluent.io/blog/stories-front-lessons-learned-supporting-apache-kafka/]
  - Under-replicated Partitions Continue to Grow Inexplicably
  - Kafka Liveness Check and Automation Causes Full Cluster Down
  - Adding a New Broker Results in Terrible Performance

• Faust (Python Streams):
@[https://github.com/robinhood/faust]
  - Faust provides both stream processing and event processing, sharing 
    similarity with tools such as Kafka Streams, Apache 
    Spark/Storm/Samza/Flink,

InfluxDB (Elas.Search Alt for TimeSeries)
@[https://logz.io/blog/influxdb-vs-elasticsearch/]

• awesome influxdb:
https://github.com/PoeBlu/awesome-influxdb
A curated list of awesome projects, libraries, tools, etc. related to InfluxDB
Tools whose primary or sole purpose is to feed data into InfluxDB.

- accelerometer2influx - Android application that takes the x-y-z axis metrics 
   from your phone accelerometer and sends the data to InfluxDB.
- agento - Client/server collecting near realtime metrics from Linux hosts
- aggregateD - A dogstatsD inspired metrics and event aggregation daemon for 
  InfluxDB
- aprs2influxdb - Interfaces ham radio APRS-IS servers and saves packet data 
  into an influxdb database
- Charmander - Charmander is a lab environment for measuring and analyzing 
  resource-scheduling algorithms
- gopherwx - a service that pulls live weather data from a Davis Instruments 
  Vantage Pro2 station and stores it in InfluxDB
- grade - Track Go benchmark performance over time by storing results in 
  InfluxDB
- Influx-Capacitor - Influx-Capacitor collects metrics from windows machines 
  using Performance Counters. Data is sent to influxDB to be viewable by grafana
- Influxdb-Powershell - Powershell script to send Windows Performance counters 
  to an InfluxDB Server
- influxdb-logger - SmartApp to log SmartThings device attributes to an 
  InfluxDB database
- influxdb-sqlserver - Collect Microsoft SQL Server metrics for reporting to 
  InfluxDB and visualize them with Grafana
- k6 - A modern load testing tool, using Go and JavaScript
- marathon-event-metrics - a tool for reporting Marathon events to InfluxDB
- mesos-influxdb-collector - Lightweight mesos stats collector for InfluxDB
- mqforward - MQTT to influxdb forwarder
- node-opcua-logger - Collect industrial data from OPC UA Servers
- ntp_checker - compares internal NTP sources and warns if the offset between 
  servers exceeds a definable (fraction of) seconds
- proc_to_influxdb - Console app to observe Windows process starts and stops 
  via InfluxDB
- pysysinfo_influxdb - Periodically send system information into influxdb (uses 
  python3 + psutil, so it also works under Windows)
- sysinfo_influxdb - Collect and send system (linux) info to InfluxDB
- snmpcollector - A full featured Generic SNMP data collector with Web 
  Administration Interface for InfluxDB
- Telegraf - (Official) plugin-driven server agent for reporting metrics into 
  InfluxDB
- tesla-streamer - Streams data from Tesla Model S to InfluxDB (rake task)
- traffic_stats - Acquires and stores statistics about CDNs controlled by 
  Apache Traffic Control
- vsphere-influxdb-go - Collect VMware vSphere, vCenter and ESXi performance 
  metrics and send them to InfluxDB

MFA OOSS
https://opensource.com/article/20/3/open-source-multi-factor-authentication Open source alternative for multi-factor authentication: privacyIDEA 

Chrony (NTP replacement)
@[https://www.infoq.com/news/2020/03/ntp-chrony-facebook/]
Facebook’s Switch from ntpd to chrony for a More Accurate, Scalable NTP Service

Apache Beam
• Apache Beam provides an advanced unified programming model, allowing 
  you to implement batch and streaming data processing jobs that can 
  run on any execution engine.
• Allows to execute pipelines on multiple environments such as Apache 
  Apex, Apache Flink, Apache Spark among others.

Apache Ignite
• Apache Ignite is a high-performance, integrated and distributed 
  in-memory platform for computing and transacting on large-scale data 
  sets in real-time, orders of magnitude faster than possible with 
  traditional disk-based or flash technologies.

• Can be used to dramatically increase RDBMS (SQL) Online Analytics Processing (OLAP)
  and Online Transaction Processing (OLTP).
@[https://www.gridgain.com/resources/papers/accelerate-mysql-olap-oltp-use-cases]

Apache PrestoDB
@[https://prestodb.io/]
- distributed SQL query engine originally developed by Facebook.
- running interactive analytic queries against data sources of
  all sizes ranging from gigabytes to petabytes.
- Engine can combine data from multiple sources (RDBMS, No-SQL, 
  Hadoop) within a single query, and it has little to no performance 
  degradation running. Being used and developed by big data giants 
  like, among others,  Facebook, Twitter and Netflix, guarantees a 
  bright future for this tool.
- Designed and written from the ground up for interactive 
  analytics and approaches the speed of commercial data warehouses 
  while scaling to the size of organizations like Facebook. 
- NOTE: Presto uses Apache Avro to represent data with schemas.
  (Avro is "similar" to Google Protobuf/gRPC)

Google Colossus FS
@[https://www.systutorials.com/3202/colossus-successor-to-google-file-system-gfs/]

ddbb from scratch
Writing a SQL database from scratch in Go: 3. indexes | notes.eatonphil.com !!!
https://notes.eatonphil.com/database-basics-indexes.html 

Mattermost+Discourse
CERN, cambia el uso de Facebook Workplace por Mattermost y Discourse
https://www.linuxadictos.com/cern-cambia-el-uso-de-facebook-workplace-por-mattermost-y-discourse.html 

Intel Optane DC performance
http://mikaelronstrom.blogspot.com/2020/02/benchmarking-5-tb-data-node-in-ndb.html?m=1
Through the courtesy of Intel I have access to a machine with 6 TB of Intel
Optane DC Persistent Memory. This is memory that can be used both as
persistent memory in App Direct Mode or simply used as a very large
DRAM in Memory Mode.

Slides for a presentation of this is available at slideshare.net.

This memory can be bigger than DRAM, but has some different characteristics
compared to DRAM. Due to this different characteristics all accesses to this
memory goes through a cache and here the cache is the entire DRAM in the
machine.

In the test machine there was a 768 GB DRAM acting as a cache for the
6 TB of persistent memory. When a miss happens in the DRAM cache
one has to go towards the persistent memory instead. The persistent memory
has higher latency and lower throughput. Thus it is important as a programmer
to ensure that your product can work with this new memory.

What one can expect performance-wise is that performance will be similar to
using DRAM as long as the working set is smaller than DRAM. As the working
set grows one expects the performance to drop a bit, but not in a very significant
manner.

We tested NDB Cluster using the DBT2 benchmark which is based on the
standard TPC-C benchmark but uses zero latency between transactions in
the benchmark client.

This benchmark has two phases, the first phase loads the data from 32 threads
where each threads loads one warehouse at a time. Each warehouse contains
almost 500.000 rows in a number of tables.

SOAP vs RESTfull vs ...
SOAP vs RESTfull vs JSON-RPC vs gRPC vs Drift

EventQL
eventql/eventql: Distributed "massively parallel" SQL query engine
https://github.com/eventql/eventql 

n-gram index
 "...Developed an indexing and search software, mostly in C. 
  The indexer is multi-threaded and computes a n-gram index 
  - stored in B-Trees - on hundreds of GB of data generated 
  every day in production. The associated search engine is also
  a multi-threaded, efficient C program."

- In the fields of computational linguistics and probability, an n-gram 
  is a contiguous sequence of n items from a given sample of text or 
  speech. The items can be phonemes, syllables, letters, words or base 
  pairs according to the application. The n-grams typically are 
  collected from a text or speech corpus. When the items are words, 
  n-grams may also be called shingles[clarification needed].

Genesis: end-to-end testing
Genesis: An End-To-End Testing & Development Platform For Distributed Systems
https://github.com/EntEthAlliance/genesis/branches
Outputting data to collect for later analysis is as simple as writing 
a JSON object to stdout. You can review the data we collect from each 
test on the test details page in the dashboard, build your own 
metrics and analytics dashboards with Kibana, and write your own 
custom data analysis and reports using Jupyter Notebooks.

fallacies of distributed computing
@[https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing]
The fallacies are[1]

    The network is reliable;
    Latency is zero;
    Bandwidth is infinite;
    The network is secure;
    Topology doesn't change;
    There is one administrator;
    Transport cost is zero;
    The network is homogeneous.

The effects of the fallacies
- Software applications are written with little error-handling on 
  networking errors. During a network outage, such applications may 
  stall or infinitely wait for an answer packet, permanently consuming 
  memory or other resources. When the failed network becomes available, 
  those applications may also fail to retry any stalled operations or 
  require a (manual) restart.
- Ignorance of network latency, and of the packet loss it can cause, 
  induces application- and transport-layer developers to allow 
  unbounded traffic, greatly increasing dropped packets and wasting 
  bandwidth.
- Ignorance of bandwidth limits on the part of traffic senders can 
  result in bottlenecks.
- Complacency regarding network security results in being blindsided 
  by malicious users and programs that continually adapt to security 
  measures.[2]
- Changes in network topology can have effects on both bandwidth and 
  latency issues, and therefore can have similar problems.
- Multiple administrators, as with subnets for rival companies, may 
  institute conflicting policies of which senders of network traffic 
  must be aware in order to complete their desired paths.
- The "hidden" costs of building and maintaining a network or subnet 
  are non-negligible and must consequently be noted in budgets to avoid 
  vast shortfalls.
- If a system assumes a homogeneous network, then it can lead to the 
  same problems that result from the first three fallacies.

Time Series + Spark
https://github.com/AmadeusITGroup/Time-Series-Library-with-Spark

NATS

Extracted from:
https://docs.baseline-protocol.org/baseline-protocol/packages/messaging
NATS is currently the default point-to-point messaging provider and 
the recommended way for organizations to exchange secure protocol 
messages. NATS was chosen due to its high-performance capabilities, 
community/enterprise footprint, interoperability with other systems 
and protocols (i.e. Kafka and MQTT) and its decentralized 
architecture.

https://docs.nats.io/
The Importance of Messaging
Developing and deploying applications and services that communicate 
in distributed systems can be complex and difficult. However there 
are two basic patterns, request/reply or RPC for services, and event 
and data streams. A modern technology should provide features to make 
this easier, scalable, secure, location independent and observable.
Distributed Computing Needs of Today
A modern messaging system needs to support multiple communication 
patterns, be secure by default, support multiple qualities of 
service, and provide secure multi-tenancy for a truly shared 
infrastructure. A modern system needs to include:
- Secure by default communications for microservices, edge 
  platforms and devices
- Secure multi-tenancy in a single distributed communication 
  technology
- Transparent location addressing and discovery
- Resiliency with an emphasis on the overall health of the system
- Ease of use for agile development, CI/CD, and operations, at scale
- Highly scalable and performant with built-in load balancing and 
  dynamic auto-scaling
- Consistent identity and security mechanisms from edge devices to 
  backend services

NATS is simple and secure messaging made for developers and 
operators who want to spend more time developing modern applications 
and services than worrying about a distributed communication system.
- Easy to use for developers and operators
- High-Performance
- Always on and available
- Extremely lightweight
- At Most Once and At Least Once Delivery
- Support for Observable and Scalable Services and Event/Data 
  Streams
- Client support for over 30 different programming languages
- Cloud Native, a CNCF project with Kubernetes and Prometheus 
  integrations

Use Cases
NATS can run anywhere, from large servers and cloud instances, 
through edge gateways and even IoT devices. Use cases for NATS 
include:
- Cloud Messaging
  - Services (microservices, service mesh)
  - Event/Data Streaming (observability, analytics, ML/AI)
- Command and Control
  - IoT and Edge
  - Telemetry / Sensor Data / Command and Control
- Augmenting or Replacing Legacy Messaging Systems

Serverless Architecture
· Also known as (AWS non official name) "Lambda" architecture.

· LinkedIn drops Lambda Arch to remove complexity                         [TODO]
@[https://www.infoq.com/news/2020/12/linkedin-lambda-architecture/]


OpenFaas (k8s)
@[https://www.openfaas.com/]
· Serverless Functions, Made Simple.                                  [_PM.low_code]
  OpenFaaS® makes it simple to deploy both functions and existing code to Kubernetes.

CliG Dev
@[https://clig.dev/]
Command Line Interface Guidelines

Debezium: Reacting to RDBM row changes
@[https://developers.redhat.com/blog/2020/12/11/debezium-serialization-with-apache-avro-and-apicurio-registry/]
- Debezium is a set of distributed services that captures row-level 
  database changes so that applications can view and respond to them. 
  Debezium connectors record all events to a Red Hat AMQ Streams Kafka 
  cluster. Applications use AMQ Streams to consume change events.

- See also: 
  https://developers.redhat.com/articles/2021/07/30/avoiding-dual-writes-event-driven-applications#

Apache AirFlow
https://dzone.com/articles/apache-airflow-20-a-practical-jump-start?edition=676391
- community driven platfom to programmatically author, schedule and monitor workflows. 

- It is NOT a data streaming solution. Tasks do not move data from one to the other
- Workflows are expected to be mostly static or slowly changing with workflows expected
  to look similar from a run to the next.

- "Batteries included" for lot of external apps:
    Amazon                Elasticsearch                                Opsgenie     Vertica
    Apache Beam           Exasol                                       Oracle       Yandex
    Apache Cassandra      Facebook                                     Pagerduty    Zendesk
    Apache Druid          File Transfer Protocol (FTP)                 Papermill
    Apache HDFS           Google                                       Plexus
    Apache Hive           gRPC                                         PostgreSQL
    Apache Kylin          Hashicorp                                    Presto
    Apache Livy           Hypertext Transfer Protocol (HTTP)           Qubole
    Apache Pig            Internet Message Access Protocol (IMAP)      Redis
    Apache Pinot          Java Database Connectivity (JDBC)            Salesforce
    Apache Spark          Jenkins                                      Samba
    Apache Sqoop          Jira                                         Segment
    Celery                Microsoft Azure                              Sendgrid
    IBM Cloudant          Microsoft SQL Server (MSSQL)                 SFTP
    Kubernetes            Windows Remote Management (WinRM)            Singularity
    Databricks            MongoDB                                      Slack
    Datadog               MySQL                                        Snowflake
    Dingding              Neo4J                                        SQLite
    Discord               ODBC                                         SSH
    Docker                OpenFaaS                                     Tableau
                                                                       Telegram

- See also:
  · https://github.com/BasPH/data-pipelines-with-apache-airflow

Jailer SQL "Explorer"
@[https://github.com/Wisser/Jailer]
• Jailer Database:  tool for database subsetting and relational data browsing.
• The Subsetter exports consistent, referentially intact row-sets 
  from relational databases, generates topologically sorted SQL-DML, 
  DbUnit datasets and hierarchically structured XML.
• The Data Browser allows bidirectional navigation through the 
  database by following foreign-key-based or user-defined relationships.

FEATURES
• Exports consistent and referentially intact row-sets from your 
  productive database and imports the data into your development and 
  test environment.
• Improves database performance by removing and archiving obsolete 
  data without violating integrity.
• Generates topologically sorted SQL-DML, hierarchically structured 
  XML and DbUnit datasets.
• Data Browsing. Navigate bidirectionally through the database by 
  following foreign-key-based or user-defined relationships.
• SQL Console with code completion, syntax highlighting and 
  database metadata visualization.
• A demo database is included with which you can get a first 
   impression without any configuration effort.

zstd fast lossless compression 
Storage: Release Zstandard v1.4.7
https://github.com/facebook/zstd/releases/tag/v1.4.7 
Zstandard, or zstd as short version, is a fast lossless compression 
algorithm, targeting real-time compression scenarios at zlib-level 
and better compression ratios. It's backed by a very fast entropy 
stage, provided by Huff0 and FSE library.

New Object Storage Protocol Could Mean the End for POSIX
https://www.enterprisestorageforum.com/cloud-storage/object-storage-protocol-could-mean-the-end-for-posix.html

Kafka: Removing Zookeeper Dependency
https://www.infoq.com/podcasts/kafka-zookeeper-removing-dependency/

Noclassified Identity Technologies
• Modern Authentication (OpenID Connect, Oauth, SAML 2.0)
• Microsoft EMS, Azure Log Analytics Server, Active Directory, 
• Kerberos, Tiering ModelMicrosoft PKI (HSM, Certificate Management), 
• Venafi CMSZero Trust Concept, B2B Guest Concept

Dual graph
 - Wikipedia
https://en.m.wikipedia.org/wiki/Dual_graph
https://en.m.wikipedia.org/wiki/Glossary_of_graph_theory_terms 

eDelivery Message exchange protocol
- ISO approves eDelivery Message exchange protocol as International Standard
@[https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/2020/07/24/ISO+approves+eDelivery+message+exchange+protocol+as+International+Standard]

CAI: standard to protect images
- Content Authenticity Initiative
- Today, at Adobe MAX 2019, in collaboration with The New York Times 
  Company and Twitter, we announced the Content Authenticity Initiative 
  (CAI) to develop an industry standard for digital content attribution.

Eclipse APP4MC
@[https://www.eclipse.org/app4mc/]
Eclipse APP4MC is a platform for engineering embedded multi- and 
many-core software systems. The platform enables the creation and 
management of complex tool chains including simulation and 
validation. As an open platform, proven in the automotive sector by 
Bosch and their partners, it supports interoperability and 
extensibility and unifies data exchange in cross-organizational 
projects. Multi- and Many-Core Development Process Support The 
Amalthea platform allows users to distribute data and tasks to the 
target hardware platforms, with the focus on optimization of timing 
and scheduling. It addresses the need for new tools and techniques to 
make effective use of the level of parallelism in this environment.

RedHat Node.js Ref. Arch.
@[https://github.com/nodeshift/nodejs-reference-architecture]

  Functional Components           Development                     Operations
  =====================           ===========                     ==========
  Web Framework                   Building good containers        Health Checks
  Template Engines                Static Assets                   Monitoring/Metrics
  Message Queuing                 Keeping up to date                Monitoring
  Internationalization            Code Quality                      Metrics Collection
  Accessibility                     Code Consistency                Distributed Tracing
  API Definition                    Testing                       Problem Determination
  GraphQL                           Code Coverage                 Logging
  Databases                       References to CI/CD             Rollout
  Authen+Author                   Npm                             Deployment
  Data Caching                      Npm Proxy/Priv.Registry         Containers
  Scaling and Multi-threading       Npm Publishing                  Serverless
  Consuming Services                Package Development           Load-balancing
  Node versions/images            Secure Development Process      Failure Handling
  Transactions_handling

Apache Calcite
" The foundation for your next high-performance database "

• Industry-standard SQL parser, validator and JDBC driver:
• Query optimization:
  Represent your query in relational algebra, transform using planning 
  rules, and optimize according to a cost model.
• "Any data, anywhere"
  Connect to third-party data sources, browse metadata, and optimize by 
  pushing the computation to the data.

CloudEvents
• Events are everywhere but event producers tend to describe events 
  differently and developers must constantly re-learn how to consume events.
  This also limits the potential for libraries, tooling and infrastructure
  to aide the delivery of event data across environments, like SDKs,
  event routers or tracing systems.

• CloudEvents(CNCF) specification describes event data in common formats to
  provide interoperability across services, platforms and systems.

• Core Specification:         
  CloudEvents                 v1.0.1 WIP
• Optional Specifications:            
  AMQP Protocol Binding       v1.0.1  WIP
  AVRO Event Format           v1.0.1  WIP
  HTTP Protocol Binding       v1.0.1  WIP
  JSON Event Format           v1.0.1  WIP
  Kafka Protocol Binding      v1.0.1  WIP
  MQTT Protocol Binding       v1.0.1  WIP
  NATS Protocol Binding       v1.0.1  WIP
  WebSockets Protocol Binding      -  WIP
  Protobuf Event Format       v1.0-rc1
  Web hook                    v1.0.1  WIP
        
• Additional Documentation:       
  CloudEvents Adapters              - WIP
  CloudEvents SDK Requirements      - WIP
  Documented Extensions             - WIP
  Primer                       v1.0.1 WIP
  Proprietary Specifications        - WIP
  

Search UI:
https://github.com/ProjectOpenSea/search-ui
https://www.elastic.co/enterprise-search/search-ui?size=n_6_n

• React Library for the fast development of modern, engaging search experiences
  quickly without re-inventing the wheel.
• Use it with Elastic App Search or Elastic Site Search to have a 
  search experience up and running in minutes.