[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"workflow-asset-ea054e01":3,"seo:featured-workflow:ea054e01-4ddd-11f1-9bc6-00163e2b0d79:zh":83,"workflow-related-asset-ea054e01-ea054e01-4ddd-11f1-9bc6-00163e2b0d79":84},{"id":4,"uuid":5,"slug":6,"title":7,"description":8,"author_id":9,"author_name":10,"author_avatar":11,"token_estimate":12,"time_saved":12,"model_used":11,"fork_count":12,"vote_count":12,"view_count":12,"parent_id":12,"parent_uuid":11,"lang_type":13,"steps":14,"tags":21,"has_voted":27,"visibility":17,"share_token":11,"is_featured":12,"content_hash":28,"asset_kind":29,"target_tools":30,"install_mode":34,"entrypoint":18,"risk_profile":35,"dependencies":37,"verification":43,"agent_metadata":46,"agent_fit":59,"trust":71,"provenance":80,"created_at":82,"updated_at":82},3252,"ea054e01-4ddd-11f1-9bc6-00163e2b0d79","asset-ea054e01","FairScale — PyTorch Extensions for Large-Scale Training","FairScale provides PyTorch extensions for high-performance and large-scale training, including fully sharded data parallelism, pipeline parallelism, and memory-efficient optimizers.","8a911193-3180-11f1-9bc6-00163e2b0d79","AI Open Source","",0,"en",[15],{"id":16,"step_order":17,"title":18,"description":11,"prompt_template":19,"variables":11,"depends_on":20,"expected_output":11},3815,1,"FairScale Training","# FairScale — PyTorch Extensions for Large-Scale Training\n\n## Quick Use\n```bash\npip install fairscale\npython -c \"\nimport torch\nfrom fairscale.nn import FullyShardedDataParallel as FSDP\nmodel = torch.nn.Linear(1024, 1024)\nfsdp_model = FSDP(model)\noptimizer = torch.optim.Adam(fsdp_model.parameters(), lr=1e-3)\nx = torch.randn(32, 1024)\nloss = fsdp_model(x).sum()\nloss.backward()\noptimizer.step()\nprint('FSDP training step complete')\n\"\n```\n\n## Introduction\nFairScale is a PyTorch library from Meta AI that provides primitives for training models that are too large to fit on a single GPU. It introduced Fully Sharded Data Parallel (FSDP), which shards model parameters, gradients, and optimizer states across GPUs to dramatically reduce per-device memory usage while maintaining training speed.\n\n## What FairScale Does\n- Shards model parameters across GPUs with Fully Sharded Data Parallel\n- Splits model layers across devices with pipeline parallelism\n- Reduces memory usage via activation checkpointing and offloading\n- Provides memory-efficient optimizers like Adascale and OSS\n- Enables training of billion-parameter models on commodity hardware\n\n## Architecture Overview\nFairScale wraps PyTorch modules with parallelism strategies. FSDP partitions each parameter tensor across the data-parallel group, gathering shards on demand for forward and backward passes. Pipeline parallelism splits sequential model stages across devices and uses micro-batching to keep all GPUs active. Both strategies compose with standard PyTorch training loops.\n\n## Self-Hosting & Configuration\n- Install via pip alongside PyTorch 1.8 or later\n- Wrap your model with FullyShardedDataParallel for memory-efficient training\n- Set auto_wrap_policy to automatically shard at Transformer block boundaries\n- Use Pipe for pipeline parallelism with manual or automatic stage splitting\n- Combine with activation checkpointing to further reduce memory\n\n## Key Features\n- FSDP enables training models that exceed single-GPU memory\n- Optimizer State Sharding (OSS) reduces optimizer memory by the world size\n- Pipeline parallelism with asynchronous scheduling for high throughput\n- Activation checkpointing trades compute for memory savings\n- Drop-in compatibility with existing PyTorch training scripts\n\n## Comparison with Similar Tools\n- **PyTorch FSDP (native)** — FairScale's FSDP was upstreamed into PyTorch core; FairScale remains useful for experimental features\n- **DeepSpeed** — broader optimization suite with ZeRO stages; FairScale is lighter and more PyTorch-native\n- **Megatron-LM** — designed for massive GPU clusters; FairScale is more accessible for smaller teams\n- **Accelerate** — higher-level API that can use FSDP internally; FairScale provides the lower-level primitives\n\n## FAQ\n**Q: Should I use FairScale or PyTorch native FSDP?**\nA: For stable production use, prefer PyTorch native FSDP (torch.distributed.fsdp). FairScale is useful for experimental features not yet upstreamed.\n\n**Q: How much memory does FSDP save?**\nA: FSDP can reduce per-GPU memory by the number of GPUs. With 8 GPUs, each device holds roughly 1\u002F8th of the model parameters.\n\n**Q: Can I use FairScale with a single GPU?**\nA: Some features like activation checkpointing work on a single GPU, but FSDP and pipeline parallelism require multiple GPUs.\n\n**Q: Does FairScale support mixed precision?**\nA: Yes. FairScale FSDP works with PyTorch AMP for FP16 or BF16 mixed-precision training.\n\n## Sources\n- https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Ffairscale\n- https:\u002F\u002Ffairscale.readthedocs.io\u002F","0",[22],{"id":23,"name":24,"slug":25,"icon":26},12,"Configs","config","⚙️",false,"b4bc74a63eb14b86d10d0f7f032c8f2a83ffa0daa9b52127bcdf12bffc68fb9c","skill",[31,32,33],"claude_code","codex","gemini_cli","single",{"executes_code":27,"modifies_global_config":27,"requires_secrets":36,"uses_absolute_paths":27,"network_access":27},[],{"npm":38,"pip":39,"brew":41,"system":42},[],[40],"fairscale",[],[],{"commands":44,"expected_files":45},[],[18],{"asset_kind":29,"target_tools":47,"install_mode":34,"entrypoint":18,"risk_profile":48,"dependencies":50,"content_hash":28,"verification":55,"inferred":58},[31,32,33],{"executes_code":27,"modifies_global_config":27,"requires_secrets":49,"uses_absolute_paths":27,"network_access":27},[],{"npm":51,"pip":52,"brew":53,"system":54},[],[40],[],[],{"commands":56,"expected_files":57},[],[18],true,{"target":32,"score":60,"status":61,"policy":62,"why":63,"asset_kind":29,"install_mode":34},98,"native","allow",[64,65,66,67,68,69,70],"target_tools includes codex","asset_kind skill","install_mode single","markdown-only","policy allow","safe markdown-only Codex install","trust established",{"author_trust_level":72,"verified_publisher":27,"asset_signed_hash":28,"signature_status":73,"install_count":12,"report_count":12,"dangerous_capability_badges":74,"review_status":75,"signals":76},"established","hash_only",[],"unreviewed",[77,78,79],"author has published assets","content hash available","no dangerous capability badges",{"owner_uuid":9,"owner_name":10,"source_url":81,"content_hash":28,"visibility":17,"created_at":82,"updated_at":82},"https:\u002F\u002Ftokrepo.com\u002Fen\u002Fworkflows\u002Fasset-ea054e01","2026-05-12 16:38:13",null,[85,141,190,237],{"id":86,"uuid":87,"slug":88,"title":89,"description":90,"author_id":91,"author_name":92,"author_avatar":11,"token_estimate":12,"time_saved":12,"model_used":11,"fork_count":12,"vote_count":12,"view_count":93,"parent_id":12,"parent_uuid":11,"lang_type":13,"steps":94,"tags":95,"has_voted":27,"visibility":17,"share_token":11,"is_featured":12,"content_hash":101,"asset_kind":29,"target_tools":102,"install_mode":34,"entrypoint":103,"risk_profile":104,"dependencies":106,"verification":111,"agent_metadata":114,"agent_fit":126,"trust":128,"provenance":131,"created_at":133,"updated_at":134,"__relatedScore":135,"__relatedReasons":136,"__sharedTags":140},2971,"d87368f0-4cd0-11f1-9bc6-00163e2b0d79","asset-d87368f0","MosaicML Composer — Efficient Large-Scale Model Training","A PyTorch library that accelerates neural network training with algorithmic speedup methods, multi-GPU support, and production-ready training recipes.","8a910e34-3180-11f1-9bc6-00163e2b0d79","Script Depot",18,[],[96],{"id":97,"name":98,"slug":99,"icon":100},11,"Scripts","script","📜","e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",[31,32,33],"SKILL.md",{"executes_code":27,"modifies_global_config":27,"requires_secrets":105,"uses_absolute_paths":27,"network_access":27},[],{"npm":107,"pip":108,"brew":109,"system":110},[],[],[],[],{"commands":112,"expected_files":113},[],[],{"asset_kind":29,"target_tools":115,"install_mode":34,"entrypoint":103,"risk_profile":116,"dependencies":118,"content_hash":101,"verification":123,"inferred":58},[31,32,33],{"executes_code":27,"modifies_global_config":27,"requires_secrets":117,"uses_absolute_paths":27,"network_access":27},[],{"npm":119,"pip":120,"brew":121,"system":122},[],[],[],[],{"commands":124,"expected_files":125},[],[],{"target":32,"score":60,"status":61,"policy":62,"why":127,"asset_kind":29,"install_mode":34},[64,65,66,67,68,69,70],{"author_trust_level":72,"verified_publisher":27,"asset_signed_hash":101,"signature_status":73,"install_count":12,"report_count":12,"dangerous_capability_badges":129,"review_status":75,"signals":130},[],[77,78,79],{"owner_uuid":91,"owner_name":92,"source_url":132,"content_hash":101,"visibility":17,"created_at":133,"updated_at":134},"https:\u002F\u002Ftokrepo.com\u002Fen\u002Fworkflows\u002Fasset-d87368f0","2026-05-11 08:32:09","2026-05-12 17:18:21",101.91813040142924,[137,138,139],"topic-match","same-kind","same-target",[],{"id":142,"uuid":143,"slug":144,"title":145,"description":146,"author_id":9,"author_name":10,"author_avatar":11,"token_estimate":12,"time_saved":12,"model_used":11,"fork_count":12,"vote_count":12,"view_count":147,"parent_id":12,"parent_uuid":11,"lang_type":13,"steps":148,"tags":149,"has_voted":27,"visibility":17,"share_token":11,"is_featured":12,"content_hash":151,"asset_kind":29,"target_tools":152,"install_mode":34,"entrypoint":153,"risk_profile":154,"dependencies":156,"verification":161,"agent_metadata":164,"agent_fit":176,"trust":178,"provenance":181,"created_at":183,"updated_at":184,"__relatedScore":185,"__relatedReasons":186,"__sharedTags":188},1835,"973c596d-3c92-11f1-9bc6-00163e2b0d79","pytorch-lightning-scalable-deep-learning-framework-973c596d","PyTorch Lightning — Scalable Deep Learning Framework","A lightweight PyTorch wrapper that decouples research code from engineering boilerplate, enabling reproducible training with automatic distributed training, mixed precision, and checkpointing.",79,[],[150],{"id":23,"name":24,"slug":25,"icon":26},"222039fca4be0c60580bda20aafc87834928ca8ef5a0bc615f055ab6716de292",[31,32,33],"PyTorch Lightning Overview",{"executes_code":27,"modifies_global_config":27,"requires_secrets":155,"uses_absolute_paths":27,"network_access":27},[],{"npm":157,"pip":158,"brew":159,"system":160},[],[],[],[],{"commands":162,"expected_files":163},[],[153],{"asset_kind":29,"target_tools":165,"install_mode":34,"entrypoint":153,"risk_profile":166,"dependencies":168,"content_hash":151,"verification":173},[31,32,33],{"executes_code":27,"modifies_global_config":27,"requires_secrets":167,"uses_absolute_paths":27,"network_access":27},[],{"npm":169,"pip":170,"brew":171,"system":172},[],[],[],[],{"commands":174,"expected_files":175},[],[153],{"target":32,"score":60,"status":61,"policy":62,"why":177,"asset_kind":29,"install_mode":34},[64,65,66,67,68,69,70],{"author_trust_level":72,"verified_publisher":27,"asset_signed_hash":151,"signature_status":73,"install_count":12,"report_count":12,"dangerous_capability_badges":179,"review_status":75,"signals":180},[],[77,78,79],{"owner_uuid":9,"owner_name":10,"source_url":182,"content_hash":151,"visibility":17,"created_at":183,"updated_at":184},"https:\u002F\u002Ftokrepo.com\u002Fen\u002Fworkflows\u002Fpytorch-lightning-scalable-deep-learning-framework-973c596d","2026-04-20 16:26:12","2026-05-12 11:43:56",91.85463498048792,[137,138,139,187],"same-author",[25,189],"configs",{"id":191,"uuid":192,"slug":193,"title":194,"description":195,"author_id":9,"author_name":10,"author_avatar":11,"token_estimate":12,"time_saved":12,"model_used":11,"fork_count":12,"vote_count":12,"view_count":196,"parent_id":12,"parent_uuid":11,"lang_type":13,"steps":197,"tags":198,"has_voted":27,"visibility":17,"share_token":11,"is_featured":12,"content_hash":200,"asset_kind":29,"target_tools":201,"install_mode":34,"entrypoint":202,"risk_profile":203,"dependencies":205,"verification":210,"agent_metadata":213,"agent_fit":225,"trust":227,"provenance":230,"created_at":232,"updated_at":233,"__relatedScore":234,"__relatedReasons":235,"__sharedTags":236},2285,"5a1c1b25-42dc-11f1-9bc6-00163e2b0d79","torchvision-computer-vision-models-datasets-transforms-5a1c1b25","torchvision — Computer Vision Models, Datasets & Transforms for PyTorch","The official PyTorch computer vision package providing pretrained models, common datasets, and image transformation utilities for training and evaluation.",65,[],[199],{"id":23,"name":24,"slug":25,"icon":26},"1cef951a2ba9146736b1907261c44e1929b268675f536152d3b3c100e2d41207",[31,32,33],"torchvision Overview",{"executes_code":27,"modifies_global_config":27,"requires_secrets":204,"uses_absolute_paths":27,"network_access":27},[],{"npm":206,"pip":207,"brew":208,"system":209},[],[],[],[],{"commands":211,"expected_files":212},[],[202],{"asset_kind":29,"target_tools":214,"install_mode":34,"entrypoint":202,"risk_profile":215,"dependencies":217,"content_hash":200,"verification":222},[31,32,33],{"executes_code":27,"modifies_global_config":27,"requires_secrets":216,"uses_absolute_paths":27,"network_access":27},[],{"npm":218,"pip":219,"brew":220,"system":221},[],[],[],[],{"commands":223,"expected_files":224},[],[202],{"target":32,"score":60,"status":61,"policy":62,"why":226,"asset_kind":29,"install_mode":34},[64,65,66,67,68,69,70],{"author_trust_level":72,"verified_publisher":27,"asset_signed_hash":200,"signature_status":73,"install_count":12,"report_count":12,"dangerous_capability_badges":228,"review_status":75,"signals":229},[],[77,78,79],{"owner_uuid":9,"owner_name":10,"source_url":231,"content_hash":200,"visibility":17,"created_at":232,"updated_at":233},"https:\u002F\u002Ftokrepo.com\u002Fen\u002Fworkflows\u002Ftorchvision-computer-vision-models-datasets-transforms-5a1c1b25","2026-04-28 16:29:19","2026-05-12 19:19:27",91.72931590331281,[137,138,139,187],[25,189],{"id":238,"uuid":239,"slug":240,"title":241,"description":242,"author_id":9,"author_name":10,"author_avatar":11,"token_estimate":12,"time_saved":12,"model_used":11,"fork_count":12,"vote_count":12,"view_count":243,"parent_id":12,"parent_uuid":11,"lang_type":13,"steps":244,"tags":245,"has_voted":27,"visibility":17,"share_token":11,"is_featured":12,"content_hash":247,"asset_kind":29,"target_tools":248,"install_mode":34,"entrypoint":249,"risk_profile":250,"dependencies":252,"verification":257,"agent_metadata":260,"agent_fit":272,"trust":274,"provenance":277,"created_at":279,"updated_at":280,"__relatedScore":281,"__relatedReasons":282,"__sharedTags":283},1939,"b2a4ac4b-3e25-11f1-9bc6-00163e2b0d79","timm-pretrained-vision-models-layers-pytorch-b2a4ac4b","timm — Pretrained Vision Models and Layers for PyTorch","timm (PyTorch Image Models) is a collection of pretrained image classification models, layers, utilities, and training scripts maintained by Ross Wightman and hosted on Hugging Face.",63,[],[246],{"id":23,"name":24,"slug":25,"icon":26},"a86c3e5d1ce7d77f20196a89d5baf26d1605e327e79ee0583c4abdf55b25d276",[31,32,33],"timm Vision Models",{"executes_code":27,"modifies_global_config":27,"requires_secrets":251,"uses_absolute_paths":27,"network_access":27},[],{"npm":253,"pip":254,"brew":255,"system":256},[],[],[],[],{"commands":258,"expected_files":259},[],[249],{"asset_kind":29,"target_tools":261,"install_mode":34,"entrypoint":249,"risk_profile":262,"dependencies":264,"content_hash":247,"verification":269},[31,32,33],{"executes_code":27,"modifies_global_config":27,"requires_secrets":263,"uses_absolute_paths":27,"network_access":27},[],{"npm":265,"pip":266,"brew":267,"system":268},[],[],[],[],{"commands":270,"expected_files":271},[],[249],{"target":32,"score":60,"status":61,"policy":62,"why":273,"asset_kind":29,"install_mode":34},[64,65,66,67,68,69,70],{"author_trust_level":72,"verified_publisher":27,"asset_signed_hash":247,"signature_status":73,"install_count":12,"report_count":12,"dangerous_capability_badges":275,"review_status":75,"signals":276},[],[77,78,79],{"owner_uuid":9,"owner_name":10,"source_url":278,"content_hash":247,"visibility":17,"created_at":279,"updated_at":280},"https:\u002F\u002Ftokrepo.com\u002Fen\u002Fworkflows\u002Ftimm-pretrained-vision-models-layers-pytorch-b2a4ac4b","2026-04-22 16:31:45","2026-05-12 15:49:56",91.70926996097583,[137,138,139,187],[25,189]]