Phillip Street

Soon after the Club was opened it became apparent that the Elizabeth Street premises were too small, and it was decided at a general meeting held at the end of 1918 to purchase a site and build new premises. By 1 September 1919, the Club had 900 members, and distinguished visitors to the Club had included the Governor-General Sir Ronald Ferguson, the State Governor Sir Walter Davidson, and the Admiral of the Fleet Viscount Jellicoe.

The Committee purchased three buildings covering a 62 ft frontage at 140, 142 and 144 Phillip Street, and produced plans to remodel the front portion, obtaining a total of four floors and a basement for a total cost estimated, including purchase, at £36,000. Of this, it was necessary to raise £11,600 by direct appeal to the members. The Club President, Commander RS Lambton, sent out a letter describing the situation as "almost a matter of 'life or death' – 'club or no club'".

Barrack Street

The results of the appeal for funds are unrecorded, other than a list of donations including £2000 from the State government and £400 from the Australian Jockey Club. It is uncertain whether the Phillip Street premises were ever occupied. (If they were, it is an interesting coincidence that the ISC would have been a near neighbour of RACA, which occupied 132–134 Phillip Street at this time.) Certainly by the 1930s the Imperial Service Club was operating from what were to be long-held rooms in Barrack Street. One of Sydney's shortest streets, Barrack Street once formed the side gate of the George Street military barracks.

The Club was situated on several floors above Sydney's first Savings Bank, the Savings Bank of New South Wales, later the Government Savings Bank. Barrack Street itself survived potential demolition under Dr. Bradfield's plans to extend a 100 ft-wide Martin Place to York Street as part of Sydney's redesign in preparation for the new Harbour Bridge.

The Harbour Bridge provides another ISC connection; the de Groot incident is alleged to have been planned in the ISC's Barrack Street rooms, according to Sydney historian Shirley Fitzgerald. On 19 March 1932, Francis Edward de Groot, described by the Sydney Morning Herald as "an officer of the New Guard", rode past Premier Jack Lang and cut the ribbon on the Bridge, shouting "I declare this bridge open on behalf of the respectable citizens of New South Wales". He was arrested and charged with "being deemed to be insane" and released the next day. With Lang's nationalism anathema to the officer and commercial classes, current ISC members agree it likely that such an "officers' coup" could have originated in the city's primary meeting place for commissioned servicemen.

The Club became the venue of choice for Battalion reunions and annual dinners. Links were maintained with forces in action during the Second World War. In 1942, when the crucial role of the 2/17th Battalion in the Battle of El Alamein was learned, ISC members from the 1/17th sent a message of congratulations to their successors. This message was said to have "tickled them most of all".

York Street

The Imperial Service Club's next premises were in York Street. Facilities included a small gymnasium, a steam and sauna room, billiards tables, and public and private dining rooms. Characters still fondly remembered by members today included Betty the cashier, who "knew everyone and everything", and Rupert the waiter, who was kept on even when he "got the shakes" in later life. Celebrated regulars included Brigadier Sir F 'Black Jack' Galleghan, who had commanded AIF troops in Changi and later became deputy head of the Commonwealth Investigation Service (which evolved from the wartime Military Intelligence Bureau). He could be seen most lunchtimes in the "always full" dining room, enjoying his sandwich and cup of tea. Newly commissioned young officers would usually be invited to York Street by other officers for a celebratory drink.

The York Street premises proved the Club's downfall. A fixed rent had been negotiated, but a vital clause in the contract was overlooked, allowing the raising of rent to market rates after a fixed period. This came into effect in 1984/5, almost doubling the Club's rent.

In April 1985 the Sydney Morning Herald reported a proposal by the ISC to the Union Club in Bent Street regarding a merger. The article detailed the Club's declining membership accompanied by escalating average fees, from a high of 4145 members and average fees of $31 in 1974, to 1574 members and fees of $260 in 1985. "Each day brings us nearer to the inevitable liquidation", said ISC President Colonel Browne. The Club's problems were attributed to "declining membership, coupled with the effect of inflation, random breath testing and changing community attitudes towards clubs". It also noted the Club's reduction to only "dining facilities and a small bank of poker machines".