#Instagram archiver archive#
#Instagram archiver full#
#Instagram archiver download#
#Instagram archiver free#

The current hang up is finding a quick way to scrape user ids from instagram, I made the list above the very slow way by having an account and following 7500 (the limit) users and then scraping them from my own account, the process is slow because ig blocks you temporarily if you follow a large number of users in a short amount of time.

#Instagram archiver full#

UPDATE: Full steam ahead, nothing holding us back.ДЄ huge thank you to those donating storage, bandwidth and code to this project. Thank you to those donating storage boxes for this project! Goals of the project or why?ДЄs mentioned above I want to grab any and all content, however something interesting to note is that generally Google isn't caching all ig accounts so often images of girls from ig are used to catfish people online, I'd like to build a database of the images we grab that is searchable by image/filename much like Google reverse image search and /u/4_pr0n i.rachives ( code available here) in order to maintain another tool that works against the creepy catfishing folk.

Nobody wants to store 100's of terabytes themselves and with no foreseeable end or timescale on this project we will be pushing the tars (none searchable) up to in the hopes they don't mind :3 I'll be managing the items and keeping them between 5-800GB each, a generous user in our irc will be helping me in that effort. This process could be streamlined but other than that it works. Using parallel my traffic looks like this, on this spec hardware I could push it to 24 jobs at once but this machine was running other tasks. I'm running 18 jobs on an 8 core mid range xeon with 32GB ram, ram overhead isn't bad using around 3GB and the cores sit at 50-70% load once it gets going.

#Instagram archiver download#

To speedup your downloads take note of -jobs 3 in rip-parallel.sh this is how many instances of ripme.jar will be spawned to download the instagram accounts. rip-parallel.sh 1st_followers_list_2mil.txt (example list) UPDATE: We're now using a new user list format, our new tool (to be released) outputs the user list one user per line without prefixing so now we use rip-parallel.sh and pass it the lists like so.

- tar.sh ~ tars the user directories, removes original files, adds current date to the filename.

- rip-parallel.sh ~ Allows us to pass user lists to ripme using parallel to speed up the downloading process.

This is the way I'm currently getting the data. I'm open to new ideas if anyone has a better tool in mind that can get the post meta data as well. I'm using RipMe 1.4.1 to get the accounts, this is a very reliable java tool newly maintained by /u/metaprime and originally wrote by /u/4_pr0n the problem with this tool while it reliably downloads the images and videos it doesn't collect the meta data of the posts (post time, location, caption, tags, comments). If you're a serious archiver and have something to offer please read on.

I've allocated 300TB on my local network for storage and post processing of the data. UPDATE: After running this project for awhile and coding ways around the initial username scraping issues we can now scrape 2 million usernames in 24 hours, so not half bad.

#Instagram archiver archive#

While some of you will only want to archive certain pages this will only serve to hinder and slow the project over time. In talks with other archivers what gets archived has been a concern of theirs, I've archived accounts containing animals, girls, cars, tattoos, etc, etc. I've made a start and have 3400* accounts 2236271 files 633GB downloaded already. To archive instagram accounts with zero prejudice. Claims made against this post were only made because I upset a user here, see this comment for details. Just make sure to tag the post with the flair and give a little background info/context. On Fridays we'll allow posts that don't normally fit in the usual data-hoarding theme, including posts that would usually be removed by rule 4: вЂњNo memes or 'look at this 'вЂќ

We are not your personal archival army.

No unapproved sale threads or advertisement posts.

#Instagram archiver free#

No memes or 'look at this old storage medium/ connection speed/purchase' (except on Free Post Fridays).

Search the Internet, this subreddit and our wiki before posting.

And we're trying really hard not to forget.ДЈ.3v Pin Reset Directions :D / Alt Imgur link Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Timetm). government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Among us are represented the various reasons to keep data - legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g.