Angular PWA - only fallback back to cached index.html if offline

I added PWA to my Angular app recently but started noticing that when serving from the sericeworker it was actually slower than not, so I looked in network tab and saw:

Whereby it was loading the first doc by serviceworker (at top) and then loading the doc 3 more times each time with a new request ID proving it was literally loading my app 3 times over...

This actually creates a noticeable pause when loading the site.

Is there any way I can make the PWA only fallback to serviceworker for index.html when offline, and just query for latest version on the spot?

Here is my ngsw-config.json:

{"$schema": "./node_modules/@angular/service-worker/config/schema.json","index": "/index.html","assetGroups": [    {"name": "app","installMode": "lazy","updateMode": "lazy","resources": {"files": ["/favicon.ico","/index.html","/*.css","/*.js"        ]      }    },    {"name": "assets","installMode": "lazy","updateMode": "lazy","resources": {"files": ["/assets/**","/*.(svg|cur|jpg|jpeg|png|apng|webp|avif|gif|otf|ttf|woff|woff2)"        ]      }    }  ]}

I also have index.html cache age set to 24 hours in my CDN.