Converting import statements to absolute in PyCharm

[Eclair_de_XII]

TL;DR Summary

I have this project in PyCharm that I have been trying to execute from the Python terminal. The problem is that the module I am using imports from other folders with import statements that act as though those modules reside in the root folder.

i.e.
I have many modules to import: [/folder/module1.py],...,[/folder/module105.py]
The blank __init__.py is assumed to be in /folder.
module: [/main.py] imports the former with the statement: [import module1],...,[import module105]

I sort of messed up while writing all my code, and now I regret it. Is there any way to fix all my import statements without manually going through every single one of the modules I created and appending the folder names to the start of each and every imported module name? I am unable to find any answers online, to my chagrin.

 

[jedishrfu]

Not understanding what you actually did to mess things up, there's a tool called fiximports that can reorganize them:

https://pypi.org/project/fiximports/

I've never used it before so you're on your own. As with any new tool usage, I would recommend backing up what you have before you use it on your files.

It's likely that what you've done can only be fixed by manually undoing it with the help of PyCharm and you'll have to decide that.

Were you attempting to make your program load faster? A major complaint with python is that it loads way more than it uses and that its hard to reduce the load without a lot of extracting and reorganizing stuff some of which you can't because its in someone else's module.

 

[Eclair_de_XII]

Were you attempting to make your program load faster?

Actually, I just was not knowledgeable on the syntax of importing modules in Python. PyCharm just allowed me to import stuff from my modules as if they were all in the root folder. Thanks for the suggestion, besides. I'll look at it later and see if it is what I am looking for. I'll keep looking for other solutions, besides.

 

[pbuk]

Is there any way to fix all my import statements without manually going through every single one of the modules I created and appending the folder names to the start of each and every imported module name?

You could write a program to do it all for you. Python would be the ideal language

 

[jedishrfu]

I see I'm not the only to write programs that write and modify programs. It must be a rite of passage for programmer programmers.

 

[Mark44]

I see I'm not the only to write programs that write and modify programs.

But can you write a program that modifies itself?

 

[jedishrfu]

Yes, I did that with the TEX programming language once. TEX allowed a program to write a file and to then execute it as code. TEX was based on a line editor command set with added variables, and control constructs created by Honeywell in the mid 1970s.

I was trying to develop a text game at the time and thought that would be pretty cool to change the rules of the game dynamically as the player played the game but then got more interested in intricacies of the task and not the game.

https://en.wikipedia.org/wiki/Text_Executive_Programming_Language

The example code in the TEX article (as was the article itself) was written by me when I was an Explorer Scout mentor:

Notable features[edit]
The most notable feature in TEX was its SUBS mode allowing variable values to crossover and become executable code. It allowed a programmer to create new variables on the fly to be used in later TEX expressions in a LISP-like fashion. TEX also allowed programmers to create scripts on the fly via line editing, saving the content to file later to be executed as part of the current program using interfile call and goto statements. However, in most cases, these features were used to provide simple dynamic goto statements in code as seen in the Adventure game parser example. What other kinds of Artificial Intelligence constructs could be developed were never fully explored.

An example of creating variables on the fly and then using them to do an interfile goto.

    _ incorporate x,y,z into the global variable pool
cmd="x=1 y=2 z=3"
subs ?
?cmd?

_ next we modify mycat/mypgm_1_2 to say "hello world" we are writing some code to execute later in our script
old mycat/mypgm_1_2
r:*cl:/!label_3 out:'Hello World'/
resave mycat/mypgm_1_2

_ lastly we subs in x,y,z and then evaluate the goto mypgm_1_2!label_3 which does an interfile goto
goto mycat/mypgm_?x?_?y?!label_?z?

The TEX program above illustrates dynamic script creation and then execution via substitution, file editing and interfile gotos. In effect, programs writing programs were possible although seldom done. In the above example, the mycat/mypgm_1_2 file would be executed at label_3 printing out "hello world".

I really liked the language, Honeywell had developed an extensive test suite for their timesharing commands in it and I was tasked with adding some of the GE extensions to the mix. In the end though GE mgmt never really required it to verify system operation from release to release.

 

[Eclair_de_XII]

Okay, this ought to do the trick.

from os import walk

source=input("source")
destination=input("destination")

#   Folders to exclude
excluded=('.idea','__pycache__','VIEW HISTORY','compiled')

name_to_prefix={}
search_in=[]

first=True

for stuff in walk(source):
    path=stuff[0]; files=stuff[-1]
    ls=path.split('\\')
    tail=ls[-1]
    if tail in excluded:
        continue
    tail+='.'
    midtail=ls[-2]+'.'
    if first:
        first=False
        item1=tail
        path1=path
    if midtail in name_to_prefix.values():
        if midtail!=item1:
            tail=midtail+tail
    for file in files:
        if file[-3:] == '.py':
            module=file[:-3]
            if module == tail[:-1]:
                continue
            search_in+=[(path,file)]
            if item1!=tail:
                tail=item1+tail
#           if tail == item1:
#               continue
#         Edited [2020-09-16 | 4:48 PM HST]
            name_to_prefix[module]=tail

old_to_new=[]

for key,val in name_to_prefix.items():
    old_to_new+=[('from '+key,'from '+val+key)]
  
def copy_replace(src,dest,strings=old_to_new):
    '''
    :param src: Path to source text file
    :param dest: Path to destination
    :param strings: list of 2-tuples
    :return: None
    '''
    original=tuple()
    with open(src,'r') as r_file:
        for line in r_file.readlines():
            original+=line,

    new=tuple()
    for line in original:
        l=line
        for tpl in strings:
            l=l.replace(tpl[0],tpl[-1])
        new+=l,

    l=dest.split('\\')[:-1]
    s='\\'.join(l)
    from os.path import exists
    from os import mkdir
    if not exists(s):
        mkdir(s)
    with open(dest,'w') as w_file:
        for line in new:
            w_file.write(line)def list_diff(biglist,smalllist):
    if len(biglist)<len(smalllist):
        biglist,smalllist=smalllist,biglist
    l=[]
    for m in biglist:
        if m not in smalllist:
            l+=[m]
    return ldef proper_import(src=source,dest=destination,files=search_in):
    '''
    :param dest: Where to save the stuff
    :param files: List of 2-tuples of str objects
    :return: None
    '''
    for path,file in files:
        l=dest.split('\\')
        s=src.split('\\')
        p=path.split('\\')
        branch=list_diff(p,s)
        if len(branch) > 0:
            l='\\'.join(l+branch)
        else:
            l='\\'.join(l)
        copy_replace(src=path+'\\'+file,dest=l+'\\'+file)

if __name__=='__main__':
    proper_import()

 

[jedishrfu]

Looks good but only testing will tell. Also its written exclusively for windows OS with the "\" path separators everywhere. Consider using os.path.sep:

# expect to see a / on Linux and MacOS and a \ on Windows
print(path.sep)

# alternatively there are functions that handle it for you in os.path
path.join(path.sep,'xxx','yyy','zzz')    # returns /xxx/yyy/zzz
path.split('/xxx/yyy/zzz')               # returns('xxx', 'yyy', 'zzz')

The best written programs of keyboard and mouse often go awry.

 

[Eclair_de_XII]

Aye. Duly noted. This code should fix all 'from ... import ...' import statements for user-defined modules. I personally never used the 'import' format.

from os import walk
from os.path import sep

#    The path where "paths.txt" is located, including "paths.txt"
#    First line should contain path to source, second should have path to destination
filename=''

k=0

with open(filename,'r') as r_file:
    for line in r_file:
        if k >1:
            break
        line=line.strip()
        if k==0:
            source=line
        else:
            destination=line
        k+=1

oldname=source.split(sep)[-1]
newname=destination.split(sep)[-1]

#   Folders to exclude
excluded=('.idea','__pycache__','VIEW HISTORY','compiled')

name_to_prefix={}
search_in=[]

first=True

for stuff in walk(source):
    path=stuff[0]; files=stuff[-1]
    ls=path.split(sep)
    tail=ls[-1]
    if tail in excluded:
        continue
    tail+='.'
    midtail=ls[-2]+'.'
    if first:
        first=False
        item1=tail
    if midtail in name_to_prefix.values():
        if midtail!=item1:
            tail=midtail+tail
    for file in files:
        if file[-3:] == '.py':
            module=file[:-3]
            if module == tail[:-1]:
                continue
            search_in+=[(path,file)]
            name_to_prefix[module]=tail

old_to_new=[]

for key,val in name_to_prefix.items():
    repl_val='from '+ val + key
    if oldname == repl_val[5:5+len(oldname)] or ' '+oldname in repl_val:
        repl_val=repl_val.replace(oldname,newname)
    else:
        u=repl_val.split(' ')
        u.insert(1,newname+'.')
        mod=[u[1]+u[2]]
        repl_val=' '.join([u[0]]+mod)
    n = len(newname)
    old_to_new += [('from ' + key, repl_val),('from '+oldname+repl_val[5+n:],repl_val)]

def copy_replace(src,dest,strings=old_to_new,replace=True):
    '''

    :param src: Path to source text file
    :param dest: Path to destination
    :param strings: list of 2-tuples
    :param replace: Indicates whether or not to replace
    :return: None
    '''
    original=tuple()
    with open(src,'r') as r_file:
        for line in r_file.readlines():
            original+=line,

    new=tuple()
    for line in original:
        l=line
        for tpl in strings:
            l=l.replace(tpl[0],tpl[-1])
        new+=l,

    l=dest.split(sep)[:-1]
    s=sep.join(l)
    from os.path import exists
    from os import mkdir

    ls=original
    if replace:
        ls=new

    if not exists(s):
        mkdir(s)
    with open(dest,'w') as w_file:
        for line in ls:
            w_file.write(line)def list_diff(biglist,smalllist):
    l=[]
    for m in biglist:
        if m not in smalllist:
            l+=[m]
    return ldef fix_imports(src=source,dest=destination,files=search_in,replace=True):
    '''
    :param src: Where the module is being copied from
    :param dest: Where the module is being copied to
    :param files: List of 2-tuples of str objects
    :param replace: Indicates whether you wish to replace import names
    :return: None
    '''
    for path,file in files:
        l=dest.split(sep)
        s=src.split(sep)
        p=path.split(sep)
        branch=list_diff(p,s)
        if len(branch) > 0:
            l=sep.join(l+branch)
        else:
            l=sep.join(l)
        copy_replace(src=path+sep+file,dest=l+sep+file,replace=replace)if __name__=='__main__':
    fix_imports(replace=True)