Five years to the day after his arrest in Cuba on espionage charges, former U.S. contractor Alan Gross is threatening a hunger strike, refusing almost all visitors and predicting he will die in prison if he isn’t freed by his 66th birthday in May, relatives and backers said Wednesday.

It’s impossible to measure the gravity of Gross’ threat, but it’s clear he is essential to any detente between Cuba and the United States. His declarations have added to a sense that the next five months could be a closing window for the Obama administration and Cuba to move to normalize their relationship.

In his first term, Obama loosened restrictions on Cuban-American travel and money remittances to Cuba, and he has advocated further changes in his second. Now, with host Panama having invited President Raul Castro to become the first Cuban leader to attend the Summit of the Americas, an annual meeting of Western Hemisphere nations, many see the time leading up to the April 10-11 meeting as a time for any U.S. action on Cuba.

The U.S. keeps Cuba under economic embargo and lists it as a state sponsor of terrorism. U.S. prisons also still hold three of five Cuban intelligence agents given long prison sentences after being convicted for operations on American soil — a topic of constant, outraged commentary in state-controlled Cuban media.

On Wednesday, the White House called on Cuba to release Gross, with press secretary Josh Earnest saying in a press release that the U.S. remains “deeply concerned” about the American’s health.

Earnest said Gross’ release “would remove an impediment to more constructive relations” between the two countries.

Cuban officials have linked the fate of the agents to Gross, who was detained in December 2009 while setting up illegal Internet access as a subcontractor for the U.S. Agency for International Development.

“There seems to be a growing sense in this country that resolving both situations would be constructive,” said Richard Klugh, a lawyer for two of the jailed Cuban agents, Antonio Guerrero and Gerardo Hernandez. “That is an atmospheric change that gives one hope that the political will is there to follow through.”

As he recently did on immigration, Obama could move without congressional approval to relax U.S. rules that require most Americans wanting to visit Cuba to go on expensive, organized trips with U.S.-approved agendas. Such a change could generate hundreds of millions of dollars a year for Cuba’s centrally planned economy, which is struggling for cash in the absence of major expansions in foreign investment or private economic activity. Cuban authorities this week sharply downgraded their prediction of 2014 growth to 1.3 percent, nearly a point lower than expected at the beginning of the year.

Observers in both countries warn, however, that expectations of imminent progress have come and gone before without real change to relations that have been tense for more than five decades.

“We won’t make enough progress obviously until he’s home,” U.S. State Department spokeswoman Marie Harf said of Gross on Tuesday. “His continued incarceration represents a significant impediment to a more constructive bilateral relationship.”

Gross’ wife, Judy, said in a written statement that the contractor has lost more than 100 pounds, can barely walk due to chronic pain, and has lost five teeth and much of the sight in his right eye. He has begun refusing to see his wife and daughter, the new chief of the U.S. Interests Section in Havana and members of Cuba’s small Jewish community, who had been visiting him on religious holidays. Backers in the U.S. said Gross has begun discussing a hunger strike, a tactic he has used in the past.

“He hasn’t been seeing anyone for a while,” said David Prinstein, vice president of the Havana-based Jewish community association. Prinstein said he hoped Gross would accept a visit for Hannukah celebrations this month.

“We maintain our hope that he will see us and keep his Jewish faith alive, and his faith that maybe this coming year he can return to his country,” Prinstein said.

This story is compiled with information from the Associated Press.