Back To The Basics Of Data Engineering - Why SFTP Remains Essential for Companies

Every day, companies share data sets of users, patient claims, financial transactions, and more with each other. Most people might assume this would be via API. However, companies have been sharing data for decades using CSVs, TSVs, positional files, and other formats you might not be familiar with. Not via API, but SFTP. I know I wouldn’t have guessed that’s how companies send data back and forth when I was in school. If you’ve been in the industry for a while, you’ve probably come across automated SFTP jobs that do just that. You’ve also likely had to encrypt or decrypt a CSV and had to interpret a schema file with parsing instructions that—somehow—are always a bit off the first time. Now, sure, we all want to build real-time systems that use LLMs and other flashy new tools and solutions. Sometimes, that’s not what is called for. After all, companies of all sizes—even tech giants like Facebook and Airbnb—still use SFTP to share critical information for analytical purposes(as well as operational). So, let’s dig into what SFTP is and how you will likely work with it. If you want to read the full article, you can find it here- https://seattledataguy.substack.com/p/data-sharing-in-the-real-world-why Or check out my blog https://www.theseattledataguy.com/ And if you want to support the channel, then you can become a paid member of my newsletter https://seattledataguy.substack.com/subscribe Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio _____________________________________________________________ Subscribe: https://www.youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmation=1 _____________________________________________________________ About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data. *I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.