The Ticking Time Bomb of Observability Expectations
"It costs money to store logs, metrics, and traces. A hard reality that is seldom discussed is that it often costs much more than that data is worth."
Oof. I'm definitely guilty of this approach, mostly due to a fantastic VelocityConf session years ago from Etsy: if it moves, graph it; if it doesn't move, graph it anyway in case it starts moving
Today's #CitusCon video 📺 of the day is from the amazing PostgreSQL hacker (on the team at Microsoft) Melanie Plageman.
In this talk, you'll learn about IO observability & the new pg_stat_io feature for Postgres that Melanie contributed to PG16.
Melanie is a great explainer, trust me you'll like this talk!
#PostgreSQL #Postgres #CitusCon #observability #Microsoft #OpenSource #database
Amazon SNS now supports AWS X-Ray active tracing for FIFO topics 👉 You can now view traces that flow through Amazon SNS FIFO topics to Amazon SQS FIFO queues https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-sns-x-ray-active-tracing-fifo-topics/ #AWS #Observability #o11y
Missed last week’s OTel Q&A? No problem! You can catch the video recording on YouTube!👇
Thanks Iris Dyrmishi for sharing her story with us!
Hey Portugal! Will see you in a couple of hours 😀 #jNation here I come to talk about #java #observability
*** NEW *** from @Sageable ...
#Application Delivery #Observability: Critical Capabilities for Modern #Software Delivery - a Sageable Solution Brief, presented by Pulse42.io
Sageable Subscriber Library (Login Required) – Sageable https://sageable.com/research-library/
Nervous and excited about my big trip to the UK and Singapore, I fly out on Monday.
If you're at #awssummit next week or #srecon APAC the following week, come say hi!
Can’t believe it’s been three years already since we kicked off Distributed Tracing in Swift, and today it finally reached 1.0.0 🎉
I will be in New York next week and will be attending
Looking forward to make new friends and talk about everything around #Reliability and #Observability
I've been thinking a lot about learning in software development. It's central to everything we do, but it's so rare that anyone pays attention to that. At least in my experience. Sometimes people will talk about being a "learning organization." I didn't hear them talk about what that means or what it takes to accomplish.
We build complex systems. Part of being complex means that we literally cannot fully understand them, so we're always learning and relearning how they work. The easier we make that, the better. #observability and incidents are two of the best opportunities we have to learn.
Come speak at #OpenSearchCon! You have until June 9th to get your presentations in for one of our three in-person tracks.
(why not submit multiple ideas!)
📈 #Analytics, #Observability, & #Security
This can be handy 👉 AWS Chatbot supports access to Amazon CloudWatch dashboards and Logs Insights in chat channels https://aws.amazon.com/about-aws/whats-new/2023/05/aws-chatbot-cloudwatch-dashboards-logs-insights-chat-channels/ #AWS #ChatOps #Observability #o11y
Check out the 👕 I chose for work. You know what it means? Yes! I'll be speaking at #ContainerDays again this year!🥳
Looking forward to seeing everyone again in Hamburg 🇩🇪
#devops #containers #cloudnative #observability #CDS23
Wow that was something! Thanks for joining my talk about #Observability #devoxxpl
👋 I'm waiting for you in room 1 if you want to hear about #Observability! Oh and btw... This room is gigantic!!! #devoxxpl
I'm speaking at Conf42 Observability in 8 days (virtual conf). Check it out, it's free.
📢 Calling all
The #OpenSearch conference #OpenSearchCon 2023 is open for submitting talk proposals 👇
#opensource #logging #observability
This can be handy 👉 Visualize and gain insights into your VPC Flow logs with Amazon Managed Grafana https://aws.amazon.com/blogs/mt/visualize-and-gain-insights-into-your-vpc-flow-logs-with-amazon-managed-grafana/ #AWS #Networking #Observability #o11y
Wrote a blog post on understanding Metrics, Logs, Traces, Events https://last9.io/blog/understanding-metrics-events-logs-traces-key-pillars-of-observability/
You know, I say in my bio and #intro that I used to work in #tech, but I haven't really actually said very much about what that was aside from comments here and there on other people's toots. (Edit: oh, and boosts are appreciated but not requested of this.)
So, here's a brief summary: I worked on two things primarily in my former career. The first was observability (more traditionally known as monitoring) systems, and second was realtime stream processing systems. In both cases the systems were BIG. Like, big. No, bigger.
The #observability systems processed hundreds of millions of datapoints/second, and the stream processing systems did terabytes/second of data. They were both globally distributed with the core backends running on thousands of machines collecting data from fleets of millions of machines. Because of the scale of them, I used to joke that I didn't count in anything smaller than a petabyte, which was honestly more true than joke.
I did a lot of different things while working on them from core engineering (I'm even academically published despite never having attended college, which is quite point of pride for me even years later,) to product management for the user interfaces for them, to #SRE type work.
As I find ways to talk about what I did more obliquely, I hope to muse on this stuff more because I still find it really cool to think about even though I've left the industry to make porn instead.
We use #Prometheus Remote Write every day so we wrote a post on it
Streaming Aggregation vs Recording Rules
Working on my @CloudLandFestival learning sessions 🌱
#Observability for Efficient #DevSecOps Pipelines
See you soon! https://www.cloudland.org/en/home/
This can be handy 👉 Use Amazon CloudWatch Internet Monitor for greater visibility into online experiences https://aws.amazon.com/blogs/mt/use-amazon-cloudwatch-internet-monitor-for-greater-visibility-into-online-experiences/ #AWS #Observability #o11y
If you're in the APAC region or are a night owl in the Americas region, you can catch me on June 1st at 11:00 SGT (23:00 ET/20:00 PT) for the DevOps Institute's SkillUp Hours, where I'll be talking about #OpenTelemetry and #Observability, alongside Suresh GP and BMK Lakshminarayanan! !
Hey 👋 Who is going to #DevoxxPL next week? My presentation is on the first day at 11:20 in Room 2. Join me if you want to listen about #DistributedTracing #Observability cc #MicrometerIO #Java #SpringBoot
Consider logging. Henrik Bærbak Christensen removed logging from a REST endpoint and reduced energy usage by 11%.
On the other hand, it's a #sustainability-#observability tradeoff, because you don't want the logging you just removed to be the one that would help you diagnose a problem.
This article by Michael Friedrich shares insights into learning #eBPF as a new #cloudnative technology that aims to improve #Observability & #Security workflows.
More insights on #InfoQ: https://bit.ly/41YBbYk
This week on Slight Reliability I tackle the humble dashboard. How do dashboards fit into the Digital Age? Are they compatible with achieving #observability?
I also share a dashboard competition we're running at SquaredUp at the end of the episode if that's something you'd be interested in.
*** NEW *** from @Sageable
Sageable #Innovation Brief: @newrelic Grok is Real #AI for #Observability & #AIOPs, Not Just Another Algorithm
https://sageable.com/research-library/ (login req'd)
Contact @Sageable to learn more. Ask about our flexible subscriber pricing and free trial program!!
Olá 👋! Will you be attending #JNation? So will I 💪🏾! I'll be talking about #observability #Micrometerio #SpringBoot #OpenTelemetry and more! Check the conference page at https://buff.ly/3LEOWW2 #java
Advanced sampling now available in AWS Distro for OpenTelemetry 👉 Tail sampling enables to take decisions when all of the trace’s spans are completed, such as “ingest 100% of all error cases and 5% of all success cases” https://aws.amazon.com/about-aws/whats-new/2023/05/aws-distro-opentelemetry-advanced-sampling/ #AWS #Observability #o11y #OTel
👋 My #eBPF learning experience for you to learn better #observability too: New InfoQ article 🌱 Thanks for the kind review help Ben Linders 🤗
#CaseStudy – find out how the Guardian has made significant improvements to the speed at which push notifications are delivered to their app users.
The solution supporting push notification delivery utilizes a range of technologies. Learn more on #InfoQ: https://bit.ly/43cxXkO
#SoftwareArchitecture #AWSLambda #Observability #Performance
Time for #SpringFramework and #Graphql and #MicrometerIo #Observability. Brian and Rossen presenting at #SpringIo23
An alert that fires if the number of firing alerts in the past 10 minutes is greater than 0, and evaluates every 5 minutes.
binjr 3.12 is now available!
This is a pretty featureful release, whose highlight is a new indexing mode for log files that makes is easy to search for arbitrary strings of character without the need for wildcards while maintaining the same level of performance, even on very large files.
It also features several other smaller enhancements and many bug fixes: full release notes available at https://binjr.eu/download/CHANGELOG/#binjr-v3120
Hey 👋! Do you know that you can add #observability behavior to your #micrometerio observations without changing any instrumentation code? Just create an <ObservationHandler> and add it to an <ObservationRegistry>! Check the 📖 for more examples https://micrometer.io/docs/observation?utm_content=buffera58af&utm_medium=social&utm_source=bufferapp.com&utm_campaign=buffer #java
Do you like to ✏️ project documentation? 😬 It's hard to keep the docs always up to date. Want to automate that? 🤔 With #MicrometerIo you can 😁! Tell us how your observations look like, we'll do the rest 💪! Attached #SpringCloud Task example #java #documentation #observability
I have a big list of all kinds of resources to learn about #Observability, #DevOps, #ContinuousDelivery, and more on my website. I'm on a mission to get people in every speciality, including testing, collaborating to succeed in these areas. Please send me your fave links! https://lisacrispin.com/observability-continuous-delivery-devops-related-resources/
Engineer: What if we instrument our applications, and we don't need the data?
SRE: What if we don't, and then we do?
There is a marginal performance cost to instrumentation in applications using #OpenTelemetry. However, the #observability upside when you need it is far greater.
Debugging SnapStart-enabled Lambda functions made easy with AWS X-Ray 👉 You can now see separate subsegments corresponding to the Restore and Invoke phases for your Lambda function’s execution 👏 https://aws.amazon.com/blogs/compute/debugging-snapstart-enabled-lambda-functions-made-easy-with-aws-x-ray/ #AWS #Serverless #Observability #o11y
Hola 👋! Will you be attending #springio Barcelona? So will I 💪🏾! I'll be co-presenting with my good friend @TommyLudwig where we will talk about #observability #micrometerio #springboot #opentelemetry and more! Check our session here https://2023.springio.net/sessions/observability-of-your-application/ #java
Do you know that there are quite a few projects that are already using #MicrometerIo observation instrumentation 🤔? One of them is #ApacheDubbo. Check the docs for more info on how to add #observability to this project! https://dubbo.apache.org/en/docs3-v2/java-sdk/advanced-features-and-usage/observability/tracing/?utm_content=buffer6049f&utm_medium=social&utm_source=bufferapp.com&utm_campaign=buffer #java
New from @Sageable and @AndiMann:
Market Brief – 'Datadog Acquires Codiga, Continues to Shift #Observability Left'
Subscriber login required - pls contact @Sageable or @AndiMann to learn more about our flexible subscription options!
🌟 Get ready to level up your #observability game! 📊🔎 With #MicrometerIO-based observability, native images become your superpower! 🚀 Don't worry about agent-based instrumentation compatibility and say hello to seamless #metrics and #tracing! ⚡️✨ #java #graalvm https://micrometer.io/docs
Hey 👋 ! Are you migrating from #SpringBoot 2.x with #SpringCloud Sleuth to #SpringBoot 3.0 ? Configure Sleuth to be compatible with Boot 3 and you're ready to go! Check this migration guide https://github.com/micrometer-metrics/tracing/wiki/Spring-Cloud-Sleuth-3.1-Migration-Guide?utm_content=buffer30725&utm_medium=social&utm_source=bufferapp.com&utm_campaign=buffer for more information! #Java #Observability #DistributedTracing
Hey 👋 ! Are you using #SpringBoot with #MicrometerIO #tracing and you don't see any spans in your #observability backend 🤔 ? Most likely that's because your spans didn't get sampled! For development, to enable 💯 % sampling set <management.tracing.sampling.probability=1.0> ! Check the docs for more information https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/?utm_content=bufferb5981&utm_medium=social&utm_source=bufferapp.com&utm_campaign=buffer#actuator.micrometer-tracing.getting-started
Using #SpringBoot 3.0 and can't seem to have #observability work in your tests? 🤔 What you're missing is most likely the <@AutoConfigureObservability> annotation on your test! You can read more about it in the ref docs https://buff.ly/3HGIrAL #MicrometerIo #testing #Java
Hard to express what it’s like attending and hosting a truly community-centric event in-person again. Here are some highlights from Day 1 of #WTFisSRE.
I’m exhausted and exhilarated and stoked for Day 2! #observability #EventHost
This has got to be one of the funniest things I've seen in a while, and certainly in the tech space.
Hey everyone! 👋 Starting today, I'm committed to making this account more active. Get ready for more #Java content! Specifically, I'll be focusing on topics like #Observability, #DistributedTracing, and #ContractTesting. Let's learn and grow together! 🌟
#Observability is a key property of today's #software systems. We're lucky to have Abby Bangser giving a solid #introduction into this topic at #SoCraTes UK (1-4 June)
Me and @jessitron are going live on twitch talking about some #observability things...
Profanity is expected.
What's this you say? You also want a UI for OpenTelemetry? Me TOO!!!
Don't miss out on #SoCraTes UK #training day. 10 great trainers, 9 great sessions.
#BDD #TDD #TBD #Kotlin #MobProgramming #EnsembleProgramming #Observability #DateHandling #LegacyCode #CodeReading #Software
Open question: What are your biggest pains or frustrations when it comes to #observability?
Let's meet at #KubeConEU 🤗
🎯 GitLab code challenge (booth P1) with stroopwafels swag
⚡ Lightning talk: Efficient #DevSecOps Pipelines
💬 Chat about #Observability, #eBPF & AI
My availability is updated publicly: https://go.gitlab.com/dxMbWj
More proof #Observability is one of the hottest tech markets.
Easily the hottest topic for my consults w/ PE, hedge funds, investment banks etc.
At least @honeycombio is 💯 OG, not #O11Y washing. Big props to them!
Honeycomb scores $50M investment - https://techcrunch.com/2023/04/06/honeycomb-lands-50m-investment-to-expand-modern-observability-platform/
🎙️ Observability via Snapshots, with Liran Haimovitch from @rookoutlabs
My latest interview for Software Daily just went live. Liran's team just announced the release of an exciting new product for their suite of Observability tools for modern dev teams.
Give this a listen to learn about the value of Snapshots for Observability, and how it's changing the landscape of devX and feature development for modern dev teams.
"From #cloud washing to #AI washing"
Marketers keep hyping emerging tech. Seeing it with #Observability too. It's counterproductive, confusing, and destroys customers' trust.
Great insights by the excellent @DavidLinthicum via @InfoWorld
The recent trend of #observability vendors pushing the "We'll reduce your alerts by using AI" message is really annoying.
As an industry, we shouldn't be sending the same data, and having the same bad data, and using some generic machine learned algorithm decide what is important.
Helping people ask questions is what observability is about, understand their systems to make them better. Giving them better telemetry is 10x better than reducing alerts.
thanks @dnsmichi for featuring my episode on #FinOps #observability and monitoring #Kubernetes cost!
In June, I'm excited to join the German #cloudnative festival @CloudLandFestival Let's learn together! 🤗
⚡ Talk: Observability for Efficient #DevSecOps Pipelines
💡 Workshop: Learning #eBPF for better #Observability
It's so cliché to say that we're excited BUT WE'RE REALLY EXCITED to announce the agenda for Monitorama PDX 2023!!!
Hope you can join us, we have some fantastic speakers and topics and we can't wait to see everyone in person.
I’m #hiring software engineers in Dublin. Join me at DocuSign to build Smarter, Easier, Trusted Agreements
#nodejs #typescript #docker #kubernetes #azure #devops #microservices #observability #opensource #jobs
Know anyone who might be interested?
👋 🌱 🐝 🤖 🛡️⛅ 👁️ 📚 🔖 🎯 🎥 🎤
Love to learn together, and follow the Ops in Dev newsletter?
Subscribe at https://buttondown.email/opsindev.news
@lili wrote a fantastic piece about #Observability in practice
@mhausenblas announced their #opentelemetry in use podcast
@horovits dicussed FinOps Observability: Monitoring Kubernetes Cost with OpenCost
From Monitoring to Observability: eBPF Chaos recording available
Netflix time series metrics system (Atlas) and dashboard system (Lumen)
New opsindev.news #newsletter, feat. Polar Signals Let's Profile, learning #eBPF and AI/ML cont., @gitpod signing commits, #DockerHub changes, #Observability in practice, #OpenTelemetry in use, otel-desktop-viewer
Let's peek into the details 👇
💥 AWS Lambda Powertools for .NET is now generally available!
The developer library helps implement observability when running Lambda functions based on .NET 6 or above, reducing the amount of custom code.
Learn more on #InfoQ: http://bit.ly/400abaG
#CloudComputing #dotnet #serverless #AWSLambda #Observability
The getting started docs for #OpenTelemetry and #dotnet have had a major update to make it a lot easier to follow and be in line with good practices.
Do let me know if there's something missing here or something you'd like to see!
#observability related question for anyone who happens to build or run a tracing backend: what happens if I send duplicate spans?
In this article I explain the difference between #metrics, #logs, #traces, and #profiles as simply as possible (using the metaphor of a café). I also discuss the relative strengths and weaknesses of each telemetry type. https://squaredup.com/blog/metrics-vs-logs-vs-traces-vs-profiles/ #observability #sre
With the recent #datadog outage, it got me thinking. In many cases there's a less refined backup option for monitoring - e.g. falling back to CloudWatch. Do teams have the knowledge to fall back to such tools? Or is this a gap? #sre #observability