Data Pipelines with Python and PostgreSQL

Data Pipelines with Python and PostgreSQL

Data Pipelines with Python and PostgreSQL

Code :
I show how to use streaming techniques to build a data pipeline which pulls data from an external API that returns massive amounts of data, and insert it straight into a PostgreSQL data base ASAP.
The Python process that reads the huge amount of data, before inserting into Postgres is able to process the incoming chunks by use of the stream=True option on the Requests module, and the iter_content method of the Request response. The mock 3rd party API, which hosts the potentially petabytes of data, makes use of the Flask stream_with_context method.


  1. I FOUND one mistake. print(t)
    that "t" shows like t=t+t, then its shows a huge data

  2. Thank you man you help me alottttt ❣️❣️ keep growing brother

  3. What if you want to do it the other way round stream from a live updating database to the or webapplication

  4. Lol Jason Statham??? JK. Nice video and clear real-world explanation. It really help me understand this topic more clearly. Thank you!

  5. Can you upload more data pipeline videos? I'm interested in transitioning into a Data Engineering career.

  6. Hi Sean,i am getting below error i tried setting my firewall connections ,it doesn't work
    HTTPConnectionPool(host='', port=1234): Max retries exceeded with url: (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<

    urllib3.connection.HTTPConnection object at 0x03C4EE50>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it')))

  7. As junior with knowledge in Java/Python/SQL wanting to break into data engineering field this was actually my first demo project and its really well explained! Thank you!
    Btw you have awesome voice, you could be voice actor on shows like Castlevania 😀 .

  8. Hello Sean thank you so much for the amazing tutorials. I've got one error following your video could you help? for some reason i cannot connect to my data base even I've got mine running same as yours in postgres


    Traceback (most recent call last):

    File "", line 8, in <module>


    File "", line 126, in connect

    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

    psycopg2.OperationalError: FATAL: database "stream_test" does not exist

  9. Great job with the video Sean! I just love it when people explain concepts in really simple ways in a short time rather than throw bombastic words for hours and you haven't learned anything at the end of it.

  10. Hello, i want to reproduce this example but where should i pull data from, which external api ? please help

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2023 53GB - WordPress Theme by WPEnjoy