How to clone the biggest repositories like a piece of cake. (@fossasia | @Google Code-In 2017)





About the Author: (Twitter | LinkedIn | Github)

So I have been finally selected for Google Code-In 2017.
I personally loved @fossasia, and started contributing to it's enormous codebase! <3

FOSSASIA is an organization developing software applications for social change using a wide-range of technologies. Projects range from Free and Open Source software, to design, graphics and hardware. FOSSASIA also organize and participate in conferences, meetups and code camps.

Let's Get Started!


What is chat.susi.ai ? ( Github | Website )

Susi is an artificial intelligence combining pattern matching, internet data, data flow principles and inference engine principles. It will have some reflection abilities and it will be able to remember the users input to produce deductions and a personalized feed-back. Its purpose is to explore the abilities of an artificial companion and to answer the remaining unanswered questions. The SUSI.AI web chat is a front-end that is developed for web access of SUSI.

In our context, It's just a large repository!

As we all know how to clone repositories via Github CLI.

I too fired simple command :

$ git clone https://github.com/0x48piraj/chat.susi.ai.git
Cloning into 'chat.susi.ai'...
remote: Counting objects: 7413, done.
error: RPC failed; curl 56 SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed


What ?? eh ??

It usually works. Right ?


After searching for hours and reading almost all related posts in SO, I resolved this issue bit by bit.

Solution (or rather workarounds) that came to my mind are:

Turn off the compression:
git config --global core.compression 0


Use shallow clone: (git clone --depth=1 url)

git clone --depth=1 https://github.com/fossasia/susi_server.git



cd into the cloned directory: (cd cloned_directory)
cd chat.susi.ai







D
eepen the clone: (git fetch --depth=N, with increasing N) 


$ git fetch --depth=5
remote: Counting objects: 31, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 31 (delta 23), reused 12 (delta 4), pack-reused 0
Unpacking objects: 100% (31/31), done.
$ git fetch --depth=100
remote: Counting objects: 885, done.
remote: Compressing objects: 100% (394/394), done.
remote: Total 885 (delta 628), reused 727 (delta 480), pack-reused 0
Receiving objects: 100% (885/885), 792.64 KiB | 103.00 KiB/s, done.
Resolving deltas: 100% (628/628), completed with 71 local objects.
$ git fetch --depth=300
remote: Counting objects: 2881, done.
remote: Compressing objects: 100% (888/888), done.
remote: Total 2881 (delta 2062), reused 2770 (delta 1954), pack-reused 0
Receiving objects: 100% (2881/2881), 2.68 MiB | 222.00 KiB/s, done.
Resolving deltas: 100% (2062/2062), completed with 72 local objects.
$ git fetch --depth=800
remote: Counting objects: 668, done.
remote: Compressing objects: 100% (425/425), done.
Reremote: Total 668 (delta 245), reused 641 (delta 218), pack-reused 0
Receiving objects: 100% (668/668), 29.95 MiB | 160.00 KiB/s, done.
Resolving deltas: 100% (245/245), completed with 23 local objects.

Unshallow the repository: (git fetch --unshallow to download all remaining revisions.) <3


$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
Luckily, my guess work came in handy. (i.e. 800 was the last revision ❤ )





Screenshots: (done on other repositories)


Partially Downloading

Final Move


Picture Time! ❤❤❤




Comments