MastodonTech.de

Neil CraigWe (BBC) have a telemetry system to measure media streaming performance & a while back we ran an experiment to slurp off a copy of those metrics (rebuffering, media type, bit rates etc.) to my log pipeline which flows in to BigQuery. Yesterday I cranked up the sample rate to BigQuery from 25% to 100% so we have a resilient system in case of incidents for Glastonbury, Wimbledon & the Euros (plus BAU content). It's definitely bumped the log rate, maybe an extra billion lines today. <a href="https://mastodon.social/tags/BBC" class="mention hashtag" rel="nofollow noopener" target="_blank">#BBC</a> <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a>

エミリーSEレクト【プログラミング入門】SQLの基本を解説！データベース操作の学ぼう！〜VTuberと学習〜【初心者向け】<a href="https://www.emilyselect.com/%e3%80%90%e3%83%97%e3%83%ad%e3%82%b0%e3%83%a9%e3%83%9f%e3%83%b3%e3%82%b0%e5%85%a5%e9%96%80%e3%80%91sql%e3%81%ae%e5%9f%ba%e6%9c%ac%e3%82%92%e8%a7%a3%e8%aa%ac%ef%bc%81%e3%83%87%e3%83%bc%e3%82%bf/?feed_id=412&_unique_id=685361c7a98dd&Mastodon" rel="nofollow noopener" translate="no" target="_blank">https://www.emilyselect.com/%e3%80%90%e3%83%97%e3%83%ad%e3%82%b0%e3%83%a9%e3%83%9f%e3%83%b3%e3%82%b0%e5%85%a5%e9%96%80%e3%80%91sql%e3%81%ae%e5%9f%ba%e6%9c%ac%e3%82%92%e8%a7%a3%e8%aa%ac%ef%bc%81%e3%83%87%e3%83%bc%e3%82%bf/?feed_id=412&_unique_id=685361c7a98dd&Mastodon</a><a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/DB" class="mention hashtag" rel="nofollow noopener" target="_blank">#DB</a> <a href="https://mastodon.social/tags/RDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#RDB</a> <a href="https://mastodon.social/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#SQL</a> <a href="https://mastodon.social/tags/%E3%82%A8%E3%82%B9%E3%82%AD%E3%83%A5%E3%83%BC%E3%82%A8%E3%83%AB" class="mention hashtag" rel="nofollow noopener" target="_blank">#エスキューエル</a> <a href="https://mastodon.social/tags/%E3%83%87%E3%83%BC%E3%82%BF%E3%83%99%E3%83%BC%E3%82%B9" class="mention hashtag" rel="nofollow noopener" target="_blank">#データベース</a> <a href="https://mastodon.social/tags/%E3%83%97%E3%83%AD%E3%82%B0%E3%83%A9%E3%83%9F%E3%83%B3%E3%82%B0" class="mention hashtag" rel="nofollow noopener" target="_blank">#プログラミング</a> <a href="https://mastodon.social/tags/%E5%85%A5%E9%96%80" class="mention hashtag" rel="nofollow noopener" target="_blank">#入門</a> <a href="https://mastodon.social/tags/%E5%88%9D%E5%BF%83%E8%80%85" class="mention hashtag" rel="nofollow noopener" target="_blank">#初心者</a> <a href="https://mastodon.social/tags/%E5%9F%BA%E6%9C%AC" class="mention hashtag" rel="nofollow noopener" target="_blank">#基本</a> <a href="https://mastodon.social/tags/%E5%AD%A6%E7%BF%92" class="mention hashtag" rel="nofollow noopener" target="_blank">#学習</a> <a href="https://mastodon.social/tags/DB%E3%83%BBSQL%E5%9F%BA%E7%A4%8E%E3%83%BB%E5%85%A5%E9%96%80" class="mention hashtag" rel="nofollow noopener" target="_blank">#DB・SQL基礎・入門</a> <a href="https://mastodon.social/tags/DB%E3%83%BBSQL%E5%9F%BA%E7%A4%8E%E3%83%BB%E5%85%A5%E9%96%80" class="mention hashtag" rel="nofollow noopener" target="_blank">#DB・SQL基礎・入門</a> <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/DB" class="mention hashtag" rel="nofollow noopener" target="_blank">#DB</a> <a href="https://mastodon.social/tags/RDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#RDB</a> <a href="https://mastodon.social/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#SQL</a> <a href="https://mastodon.social/tags/%E3%82%A8%E3%82%B9%E3%82%AD%E3%83%A5%E3%83%BC%E3%82%A8%E3%83%AB" class="mention hashtag" rel="nofollow noopener" target="_blank">#エスキューエル</a> <a href="https://mastodon.social/tags/%E3%83%87%E3%83%BC%E3%82%BF%E3%83%99%E3%83%BC%E3%82%B9" class="mention hashtag" rel="nofollow noopener" target="_blank">#データベース</a> <a href="https://mastodon.social/tags/%E3%83%97%E3%83%AD%E3%82%B0%E3%83%A9%E3%83%9F%E3%83%B3%E3%82%B0" class="mention hashtag" rel="nofollow noopener" target="_blank">#プログラミング</a> <a href="https://mastodon.social/tags/%E5%85%A5%E9%96%80" class="mention hashtag" rel="nofollow noopener" target="_blank">#入門</a> <a href="https://mastodon.social/tags/%E5%88%9D%E5%BF%83%E8%80%85" class="mention hashtag" rel="nofollow noopener" target="_blank">#初心者</a> <a href="https://mastodon.social/tags/%E5%9F%BA%E6%9C%AC" class="mention hashtag" rel="nofollow noopener" target="_blank">#基本</a> <a href="https://mastodon.social/tags/%E5%AD%A6%E7%BF%92" class="mention hashtag" rel="nofollow noopener" target="_blank">#学習</a>

Eric HeikenGoogle Cloud Platform seems to be having issues. I'm having problems loading stuff in BigQuery and I'm getting 503s on ETLs pushing into BQ. Are you seeing problems? <a href="https://mastodon.social/tags/bigquery" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigquery</a> <a href="https://mastodon.social/tags/datascience" class="mention hashtag" rel="nofollow noopener" target="_blank">#datascience</a>

Jons MostovojsHow messy is <a href="https://social.doma.dev/tags/terraform" class="mention hashtag" rel="nofollow noopener" target="_blank">#terraform</a> with <a href="https://social.doma.dev/tags/gcp" class="mention hashtag" rel="nofollow noopener" target="_blank">#gcp</a>?I'm trying to make a system where <a href="https://social.doma.dev/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#rust</a> worker ingests data of HTTP requests via <a href="https://social.doma.dev/tags/cloudrun" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudrun</a>, and passes them into <a href="https://social.doma.dev/tags/bigtable" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigtable</a>, which uses further ingestion recipe to export the data into <a href="https://social.doma.dev/tags/bigquery" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigquery</a> I have tried to make a complete terraform declaration for this, but got into permission issues, then I have tried to make a system that generates all the artefacts like service accounts and docker image and then refers to those from terraform builds, but I almost don't see a value of doing it like that.Does anyone have an example of <a href="https://social.doma.dev/tags/cloudrun" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudrun</a> <a href="https://social.doma.dev/tags/cdc" class="mention hashtag" rel="nofollow noopener" target="_blank">#cdc</a>? I am new to this and I feel really slow. <a href="https://social.doma.dev/tags/lazyweb" class="mention hashtag" rel="nofollow noopener" target="_blank">#lazyweb</a> <a href="https://social.doma.dev/tags/askfedi" class="mention hashtag" rel="nofollow noopener" target="_blank">#askfedi</a>

Code EnigmaOur head of development, Dioni, released the first version of the <a href="https://toots.codeenigma.com/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> module for <a href="https://toots.codeenigma.com/tags/Drupal" class="mention hashtag" rel="nofollow noopener" target="_blank">#Drupal</a> last month: <a href="https://www.drupal.org/project/bigquery_integration" rel="nofollow noopener" translate="no" target="_blank">https://www.drupal.org/project/bigquery_integration</a>Hat tip to our client the Waste and Resources Action Programme (WRAP) for agreeing to share it. 😎

:rss: Qiita - 人気の記事MCPでのデータベースとの対話+資料化 <a href="https://qiita.com/jw-automation/items/ee08849048f1517d40b2?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items" rel="nofollow noopener" translate="no" target="_blank">https://qiita.com/jw-automation/items/ee08849048f1517d40b2?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items</a><a href="https://rss-mstdn.studiofreesia.com/tags/qiita" class="mention hashtag" rel="nofollow noopener" target="_blank">#qiita</a> <a href="https://rss-mstdn.studiofreesia.com/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#Python</a> <a href="https://rss-mstdn.studiofreesia.com/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://rss-mstdn.studiofreesia.com/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://rss-mstdn.studiofreesia.com/tags/MCP" class="mention hashtag" rel="nofollow noopener" target="_blank">#MCP</a> <a href="https://rss-mstdn.studiofreesia.com/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a>

SpatialistsNew geospatial data in Google BigQuery: <a href="https://mapstodon.space/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#Google</a> is adding geospatial content to its <a href="https://mapstodon.space/tags/DWH" class="mention hashtag" rel="nofollow noopener" target="_blank">#DWH</a> solution <a href="https://mapstodon.space/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a>. Additions encompass annotated Street View <a href="https://mapstodon.space/tags/imagery" class="mention hashtag" rel="nofollow noopener" target="_blank">#imagery</a>, Places (<a href="https://mapstodon.space/tags/POI" class="mention hashtag" rel="nofollow noopener" target="_blank">#POI</a>) data, and <a href="https://mapstodon.space/tags/traffic" class="mention hashtag" rel="nofollow noopener" target="_blank">#traffic</a> data, among others. <a href="https://spatialists.ch/posts/2025/04-14-new-geospatial-data-in-google-bigquery/" rel="nofollow noopener" translate="no" target="_blank">https://spatialists.ch/posts/2025/04-14-new-geospatial-data-in-google-bigquery/</a> <a href="https://mapstodon.space/tags/GIS" class="mention hashtag" rel="nofollow noopener" target="_blank">#GIS</a> <a href="https://mapstodon.space/tags/GISchat" class="mention hashtag" rel="nofollow noopener" target="_blank">#GISchat</a> <a href="https://mapstodon.space/tags/geospatial" class="mention hashtag" rel="nofollow noopener" target="_blank">#geospatial</a> <a href="https://mapstodon.space/tags/SwissGIS" class="mention hashtag" rel="nofollow noopener" target="_blank">#SwissGIS</a>

ClintonDiving into <a href="https://mastodon.social/tags/Vermont" class="mention hashtag" rel="nofollow noopener" target="_blank">#Vermont</a> wildlife for the <a href="https://mastodon.social/tags/30DayChartChallenge" class="mention hashtag" rel="nofollow noopener" target="_blank">#30DayChartChallenge</a> "circle" day! 🦌 Using <a href="https://mastodon.social/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#Python</a> & <a href="https://mastodon.social/tags/plotly" class="mention hashtag" rel="nofollow noopener" target="_blank">#plotly</a> to compare monthly <a href="https://mastodon.social/tags/Moose" class="mention hashtag" rel="nofollow noopener" target="_blank">#Moose</a> and <a href="https://mastodon.social/tags/BlackBear" class="mention hashtag" rel="nofollow noopener" target="_blank">#BlackBear</a> sightings 🐻 Data wrangled with <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> and <a href="https://mastodon.social/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#SQL</a>. Any guesses which animal is seen more consistently throughout the year? 😜 <a href="https://mastodon.social/tags/DataViz" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataViz</a> <a href="https://mastodon.social/tags/Wildlife" class="mention hashtag" rel="nofollow noopener" target="_blank">#Wildlife</a> <a href="https://mastodon.social/tags/RadialChart" class="mention hashtag" rel="nofollow noopener" target="_blank">#RadialChart</a>

Hacker NewsBigQuery pricing model cost us $10k in 22 seconds<a href="https://www.linkedin.com/posts/yingjun-wu_%F0%9D%90%81%F0%9D%90%A2%F0%9D%90%A0%F0%9D%90%90%F0%9D%90%AE%F0%9D%90%9E%F0%9D%90%AB%F0%9D%90%B2%F0%9D%90%AC-%F0%9D%90%91%F0%9D%90%88%F0%9D%90%83%F0%9D%90%88%F0%9D%90%82%F0%9D%90%94%F0%9D%90%8B%F0%9D%90%8E%F0%9D%90%94-activity-7307736315079405569-F2Ng" rel="nofollow noopener" translate="no" target="_blank">https://www.linkedin.com/posts/yingjun-wu_%F0%9D%90%81%F0%9D%90%A2%F0%9D%90%A0%F0%9D%90%90%F0%9D%90%AE%F0%9D%90%9E%F0%9D%90%AB%F0%9D%90%B2%F0%9D%90%AC-%F0%9D%90%91%F0%9D%90%88%F0%9D%90%83%F0%9D%90%88%F0%9D%90%82%F0%9D%90%94%F0%9D%90%8B%F0%9D%90%8E%F0%9D%90%94-activity-7307736315079405569-F2Ng</a><a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#HackerNews</a> <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/Cost" class="mention hashtag" rel="nofollow noopener" target="_blank">#Cost</a> #$10k <a href="https://mastodon.social/tags/DataAnalysis" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataAnalysis</a> <a href="https://mastodon.social/tags/CloudComputing" class="mention hashtag" rel="nofollow noopener" target="_blank">#CloudComputing</a> <a href="https://mastodon.social/tags/PricingModel" class="mention hashtag" rel="nofollow noopener" target="_blank">#PricingModel</a> <a href="https://mastodon.social/tags/TechNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#TechNews</a>

N-gated Hacker News🍕🎨 "Map of Python" is the digital dabbling of a cartographer, lost in the vast jungle of 500,000+ <a href="https://mastodon.social/tags/Python" class="mention hashtag" rel="nofollow noopener" target="_blank">#Python</a> packages, desperately trying to create art from JSON blobs. 🧩🤷‍♂️ But hey, at least there's <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> to save the day from the horror of actually downloading data! 🚀🧐 <a href="https://fi-le.net/pypi/" rel="nofollow noopener" translate="no" target="_blank">https://fi-le.net/pypi/</a> <a href="https://mastodon.social/tags/DataVisualization" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataVisualization</a> <a href="https://mastodon.social/tags/Cartography" class="mention hashtag" rel="nofollow noopener" target="_blank">#Cartography</a> <a href="https://mastodon.social/tags/ArtInTech" class="mention hashtag" rel="nofollow noopener" target="_blank">#ArtInTech</a> <a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#HackerNews</a> <a href="https://mastodon.social/tags/ngated" class="mention hashtag" rel="nofollow noopener" target="_blank">#ngated</a>

Recce - Trust, Verify, Ship❌ No more manually cross-referencing dbt docs from dev and prod❌ No more manually checking schemas in your data warehouse❌ No more manually comparing row-counts on models.See a trend? Read how an experienced data professional validates zero regression on a <a href="https://mastodon.social/tags/dbt" class="mention hashtag" rel="nofollow noopener" target="_blank">#dbt</a> PR:<a href="https://www.linkedin.com/posts/abdelm_recce-dbt-activity-7300436694808358914-aKiF?rcm=ACoAAAVLBgkB6r61wCsOcKgFDrtf_EEEe3UjdXs" rel="nofollow noopener" translate="no" target="_blank">https://www.linkedin.com/posts/abdelm_recce-dbt-activity-7300436694808358914-aKiF?rcm=ACoAAAVLBgkB6r61wCsOcKgFDrtf_EEEe3UjdXs</a><a href="https://mastodon.social/tags/DataEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataEngineering</a> <a href="https://mastodon.social/tags/dbt" class="mention hashtag" rel="nofollow noopener" target="_blank">#dbt</a> <a href="https://mastodon.social/tags/Data" class="mention hashtag" rel="nofollow noopener" target="_blank">#Data</a> <a href="https://mastodon.social/tags/Analytics" class="mention hashtag" rel="nofollow noopener" target="_blank">#Analytics</a> <a href="https://mastodon.social/tags/AnalyticsEngineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#AnalyticsEngineering</a> <a href="https://mastodon.social/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#SQL</a> <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenSource</a>

Stuifbergen.comGA4 intraday exports and cookieless pingsI build a lot of reports for clients that use Big Query GA4 as source.Now.. that works like a charm. But.. you will need to wait some time to get processed data from the <code>events_</code> tables.More recent data will appear in the streaming <code>_intraday_</code> tables, if you have that enabled. But.. that data is not always complete! Especially when your site has consent mode enabled, and does not set a cookie until after consent.Here’s how it works:The scenarioSomeone visits the site for the first time (source: some campaign), gets confronted with the cookie banner, and then clicks accept.We tagged the site correctly, so this is what happens<ol><li>a <code>page_view</code> event triggers (with URL parameters) – and notices analytics consent is denied (the default)</li><li>the tracker attaches some parameters to this hit, to help processing<ul><li>a session is started</li><li>this is the first visit</li></ul></li><li>there is an item list on the page: <code>view_item_list</code> event is triggered</li><li>the cookiebanner pops up (event: <code>cookiebar_view</code>)</li><li>the visitor clicks accept (event: <code>cookiebar_accept</code>) and the tracker gets sent a <code>granted</code> signal</li><li>now the cookie can be uses, and is attached to an automatic event <code>user_engagement</code></li></ol>Sounds simple. Now, let’s see what is streamed into Big Query:The streaming data gapBasically, the intraday tables store what happens, as it happens.<ul><li>cookie field ( <code>user_pseudo_id</code> ) is filled in on hits on/after consent</li><li>cookie field is <code>NULL</code> for hits before consent</li></ul>As it should be, right? But there’s a third bullet:<ul><li>first batch of events will not appear in the <code>intraday</code> table!</li></ul>Here’s what we see (most recent hit first, read from bottom to top)<ol><li>the page_view is missing in the streaming table</li><li>the <code>collected_traffic_source</code> information is missing (it is always only filled in on the first batch of events)</li><li>As a byproduct, we also do not see the session start and first visit</li><li>the other events are all sent without a cookie</li><li>after consent, we see the <code>user_pseudo_id</code> – finally</li></ol>The next day.. Google has glued it all togetherProcessed data: every event has a rowThe following is in the processed data: (most recent hit first, read from bottom to top)<ul><li>The page_view event and all other events leading up to the consent have a cookie attached to it! Google rescued that information</li><li>the “Attached” parameters to the hit expand to two extra rows<ul><li>session_start</li><li>first_visit</li></ul></li><li>we have source information: <code>collected_traffic_source</code> is present – on the first batch, as normal</li></ul>Not visible in the screenshot: <code>session_traffic_source_last_click</code> – the session information is properly filled in.The consequencesIf you decide to use <code>intraday</code> tables in your Big Query reports: be aware that although the information is fresh (no pun intended, GA360 users), it’s incomplete<ul><li>intraday misses crucial events, namely the first batch (most often a page_view)<ul><li>bye bye landing_page reports based on page_views</li><li>bye bye traffic source reports based on session_traffic_source_last_click or collected_traffic_source</li></ul></li><li>intraday misses cookies on some events<ul><li>which is not too much of an issue, really</li></ul></li></ul>Your experiences?Do you use intraday tables in your models? Have you found clever workarounds to get the correct data in?Let me know! Drop a comment here, or send me a <a href="https://bsky.app/profile/zjuul.net" rel="nofollow noopener" target="_blank">bluesky</a> message!Still here?Check out GA4Dataform – a product I’ve helped build that <a href="https://ga4dataform.com/" rel="nofollow noopener" target="_blank">turns the GA4 Big Query exports into usable tables</a>! Related posts: <a class="" rel="nofollow noopener" href="https://stuifbergen.com/2021/09/google-analytics-4-truncates-page-location/" target="_blank">Google Analytics 4 truncates page location</a><a class="" rel="nofollow noopener" href="https://stuifbergen.com/2022/01/making-sense-of-event-parameters-in-ga4/" target="_blank">Making sense of Event Parameters in GA4</a><a class="" rel="nofollow noopener" href="https://stuifbergen.com/2022/06/powertips-for-ga4-implementations/" target="_blank">Make your GA4 life easier: Some powertips!</a><a class="" rel="nofollow noopener" href="https://stuifbergen.com/2024/04/smart-incremental-ga4-tables-in-dataform/" target="_blank">Smart incremental GA4 tables in Dataform</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://stuifbergen.com/tag/big-query/" target="_blank">#bigQuery</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://stuifbergen.com/tag/consent-mode/" target="_blank">#consentMode</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://stuifbergen.com/tag/cookies/" target="_blank">#cookies</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://stuifbergen.com/tag/ga4/" target="_blank">#ga4</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://stuifbergen.com/tag/tagging/" target="_blank">#tagging</a>

Gus🚀 DataTalksClub's Data Engineering Zoomcamp Week 3 - BigQuery as a data warehousing solution.🎯 For this week's module, we used Google's BigQuery to read Parquet files from a GCS bucket, and compare querying on regular, external and partitioned/clustered tables.🔗 My answers to this module: <a href="https://github.com/goosethedev/de-zoomcamp-2025/blob/ecb1f1f3fc69b8d10703eb07328567dab2acf688/03-data-warehousing/README.md" rel="nofollow noopener" translate="no" target="_blank">https://github.com/goosethedev/de-zoomcamp-2025/blob/ecb1f1f3fc69b8d10703eb07328567dab2acf688/03-data-warehousing/README.md</a><a href="https://mastodon.social/tags/dataengineering" class="mention hashtag" rel="nofollow noopener" target="_blank">#dataengineering</a> <a href="https://mastodon.social/tags/bigquery" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigquery</a> <a href="https://mastodon.social/tags/bootcamp" class="mention hashtag" rel="nofollow noopener" target="_blank">#bootcamp</a> <a href="https://mastodon.social/tags/bash" class="mention hashtag" rel="nofollow noopener" target="_blank">#bash</a> <a href="https://mastodon.social/tags/python" class="mention hashtag" rel="nofollow noopener" target="_blank">#python</a>

JulesWe released our product as Open Source today!So if you're into Google Analytics and Big Query: check it out, it's free to use.Tags: <a href="https://mastodon.social/tags/gpl" class="mention hashtag" rel="nofollow noopener" target="_blank">#gpl</a> <a href="https://mastodon.social/tags/ga4" class="mention hashtag" rel="nofollow noopener" target="_blank">#ga4</a> <a href="https://mastodon.social/tags/bigquery" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigquery</a> <a href="https://mastodon.social/tags/measure" class="mention hashtag" rel="nofollow noopener" target="_blank">#measure</a> <a href="https://mastodon.social/tags/sql" class="mention hashtag" rel="nofollow noopener" target="_blank">#sql</a> <a href="https://mastodon.social/tags/dbt" class="mention hashtag" rel="nofollow noopener" target="_blank">#dbt</a> <a href="https://mastodon.social/tags/dataform" class="mention hashtag" rel="nofollow noopener" target="_blank">#dataform</a> <a href="https://ga4dataform.com/were-open-source-ga4dataform-community-is-released/" rel="nofollow noopener" translate="no" target="_blank">https://ga4dataform.com/were-open-source-ga4dataform-community-is-released/</a>

Nicolas Fränkel 🇺🇦🇬🇪What's the best way to wake up than to try to fix a failing GitHub Workflow, and to finally realize <a href="https://mastodon.top/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#Google</a> deleted the underlying <a href="https://mastodon.top/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> table because the default expiration date is 60 days?Can't there be a big warning when you create the table? 🤬<a href="https://mastodon.top/tags/DXFailure" class="mention hashtag" rel="nofollow noopener" target="_blank">#DXFailure</a> IMHO

PPC LandGoogle Analytics updates BigQuery Export with enhanced traffic source analysis: Learn about the latest Google Analytics BigQuery export enhancement adding session-scoped traffic source fields for deeper insights. <a href="https://ppc.land/google-analytics-updates-bigquery-export-with-enhanced-traffic-source-analysis/?utm_source=dlvr.it&utm_medium=mastodon" rel="nofollow noopener" translate="no" target="_blank">https://ppc.land/google-analytics-updates-bigquery-export-with-enhanced-traffic-source-analysis/?utm_source=dlvr.it&utm_medium=mastodon</a> <a href="https://mastodon.social/tags/GoogleAnalytics" class="mention hashtag" rel="nofollow noopener" target="_blank">#GoogleAnalytics</a> <a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/DataAnalytics" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataAnalytics</a> <a href="https://mastodon.social/tags/TrafficSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#TrafficSource</a> <a href="https://mastodon.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener" target="_blank">#DigitalMarketing</a>

Guia de TIWebinar de Google Cloud Fundamentals Gratuito da Fast Lane<a href="https://guiadeti.com.br/webinar-google-cloud-fundamentals-gratuito/?feed_id=10029&_unique_id=675709187aa29" rel="nofollow noopener" translate="no" target="_blank">https://guiadeti.com.br/webinar-google-cloud-fundamentals-gratuito/?feed_id=10029&_unique_id=675709187aa29</a><a href="https://flipboard.social/tags/bigquery" class="mention hashtag" rel="nofollow noopener" target="_blank">#bigquery</a> <a href="https://flipboard.social/tags/cloud" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloud</a> <a href="https://flipboard.social/tags/cloudcomputing" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudcomputing</a> <a href="https://flipboard.social/tags/googlecloud" class="mention hashtag" rel="nofollow noopener" target="_blank">#googlecloud</a> <a href="https://flipboard.social/tags/tecnologia" class="mention hashtag" rel="nofollow noopener" target="_blank">#tecnologia</a> <a href="https://guiadeti.com.br/webinar-google-cloud-fundamentals-gratuito/?feed_id=10029&_unique_id=675709187aa29" rel="nofollow noopener" translate="no" target="_blank">https://guiadeti.com.br/webinar-google-cloud-fundamentals-gratuito/?feed_id=10029&_unique_id=675709187aa29</a>

Joe<a href="https://functional.cafe/tags/Snowflake" class="mention hashtag" rel="nofollow noopener" target="_blank">#Snowflake</a> <a href="https://functional.cafe/tags/Terraform" class="mention hashtag" rel="nofollow noopener" target="_blank">#Terraform</a> and <a href="https://functional.cafe/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a>

Neil Craig...and after even more debugging, it turns out that the reason for BigQuery `appendRows()` write failures was that large write data volumes (~5k rows or ~5MB of data) exceed the (undocumented) default `createStreamConnection()` timeout 🤦🏻‍♂️. Even the units of the config option (once you find it) are not documented 🤦🏻‍♂️🤦🏻‍♂️. It's in milliseconds as it turns out.I upped the timeout to 120s and the failures go away. FFS.<a href="https://cloud.google.com/nodejs/docs/reference/bigquery-storage/latest/bigquery-storage/managedwriter.writerclient" rel="nofollow noopener" translate="no" target="_blank">https://cloud.google.com/nodejs/docs/reference/bigquery-storage/latest/bigquery-storage/managedwriter.writerclient</a><a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/GoogleCloud" class="mention hashtag" rel="nofollow noopener" target="_blank">#GoogleCloud</a> <a href="https://mastodon.social/tags/NodeJS" class="mention hashtag" rel="nofollow noopener" target="_blank">#NodeJS</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebDev</a>

Neil CraigAfter a goodly amount of debugging, it turns out that the BigQuery Storage API (for NodeJS at least) function `appendRows()` (which is how you tell it to write data to BigQuery) fails every time if you give it "too much" data.This is not documented (and the docs are somewhat minimal for such an important function). I have an open case with Google & have asked for the docs to be improved.Thought it might help others to note this.<a href="https://cloud.google.com/nodejs/docs/reference/bigquery-storage/latest/bigquery-storage/v1.bigquerywriteclient#_google_cloud_bigquery_storage_v1_BigQueryWriteClient_appendRows_member_1_" rel="nofollow noopener" translate="no" target="_blank">https://cloud.google.com/nodejs/docs/reference/bigquery-storage/latest/bigquery-storage/v1.bigquerywriteclient#_google_cloud_bigquery_storage_v1_BigQueryWriteClient_appendRows_member_1_</a><a href="https://mastodon.social/tags/BigQuery" class="mention hashtag" rel="nofollow noopener" target="_blank">#BigQuery</a> <a href="https://mastodon.social/tags/NodeJS" class="mention hashtag" rel="nofollow noopener" target="_blank">#NodeJS</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebDev</a>

Frühere Suchanfragen

Suchoptionen

Verwaltet von:

Serverstatistik:

#bigquery