Data Engineering Ask Me Anything With Seattle Data Guy - 2024

What questions do you have about data engineering? Are you curious about different tools, skills, how can I help answer them? I haven't done a live where it's just me in a while and I wanted to open it up to y'all! 1:18 - Hey how are you? 2:29 - What's the best 'stack' for data engineering? 3:32 - What made you start this meeting at the 25th minute instead of the 30th? 3:50 - Which cloud provider should I choose, and also is easy to switching between them if we know concepts? 5:10 - Any tips for my first interview in big tech in UK? 6:30 - What niche within data engineering is most rewarding at the moment? It's better to be a generalista or specialist? 8:20 - What do we need to study to slowly learn and create architecture for project? 9:34 - I have 2.5 years as Qa Tester, can I show this experience in my data engineer resume? 10:20 - Which is best snowflake or Databricks? 12:15 - I have pivoted Towards data platform engineer from Analytics engineer. I feel like my current job is more replaceable by AI, thoughts? 13:00 - How much DSA is enough DSA for data engineers? 14:05 - I Spent 20 years in the ERP and HCM applications, and moved to the enterprise data team last month. What tips do you have for me? 15:45 - Whats easier pivoting rom backend dev to DE or the Other way around? What do you think is better to start with? 16:59 - It appears data governance and excellence are limited to the enterprise data teams. The application teams do not pay much attention to it. Any advice? 18:52 - What AWS course do you suggest for begginers? 19:42 - Can you suggest some projects tha can help us to get a data enginner job? And how to prepare a resume? 22:20 - I have just started upwork freelancing around B.I. I want to figure out how to make na offer 24:15 - How do I stand out as a DE as na applicant or as a worker in general? What makes one DE better than the Other? 26:21 - I have no background in it but completed aws sol architect asso and aws dev associate. A company hired e as aws data engineer, idk even know how to code. What do you suggest i do? 27:30 - How much domain knowledge would you say is required to get into Consulting and get a Project? 29:35 -Is there like blind 75 leetcode for sql that i need to study to prepare for a Technical interview for DE? Trying to transition from web developer to DE. 31:03 - I don't have system design skills, i have interview in coming monday, any suggestions? 33:00 - In na enterprise seeting do analysts usually have acess to the dw and query the data they need? Or the DE teams just give them flat files such as csv/parquet for them to Work on? 34:18 - For a few 10s of gb of data, do you recommend big data tools like spark or Athena?. Can we achiev the same with more relational databases ike mysql? Which one is better in terms of cost and future scale? 36:05 - I have na interview on Monday with a startup working on customer engagement like attribution and events data. Any advise on how to not blow this? 37:10 - Have you ever worked as a geospatial DE with satellite data? 37:42 - How can I learn DE, I have some background as web dev with php and mysql, but idk how to become DE 38:28 - We are implementing data contracts. We need to drive cultural change. Any tips on how to get an org to focus on data in sources. In a scandi country that has flat orgs so cant rely on mgmt 43:32 - Im a flutter dev, do you recommend me learning data sicence or should I be full stack data? 44:40 - Hi, 3 years back I just landed into DE after graduation, and directly started with snowflake and i am good with it. I want to become a complete DE, what should i do 46:24 - Got a job from chat gpt, but idk anything what to do? 49:35 - As a DE do we need to understand DSA to land Jobs? And there are barries for e to land International remote from Nigeria? even thought I have 3 years exp, what do you suggest? 51:15- DE and MLE need to know leetcode? 52:11- How do you make sure your pipeline is idempotente? We ingest Gbs of files on hourly-basis, partition of table is on day-basis. How do i make sure I ingest corrected hourly file without overwriitng daypartition? 53:30 - Why are there not more DE Jobs Towards apache superste Analytics? 55:50- Starting DE in 2025, is it Worth? 58:10 - Suggestions on changing schema? How do you change a column datatype r add a newcolumn/remove a column withtout overwriting the table? Table is in PB of size 1:00:10 - AS a entry level DE, do I need to know Kafka? 1:00:56 - Any suggestion on dealing with legacy systems? 1:01:25 - Should MLE learn Databricks or snowflake? 1:01:57 - How much did you spend upskilling that got you to where you are right now? 1:03:13 - Did you ever go through the whole imposter syndrome phase? 1:05:50 - I Work as DE and i fell like i am Always 3 to 5 years behind in current tech. What are your main sources to be up to date with your tech stack?