Difference between revisions of "Py-3rdparty-mediawiki"

From BITPlan mediawiki-japi Wiki
Jump to navigation Jump to search
Line 138: Line 138:
 
</source>
 
</source>
  
 +
= wikibackup =
 +
== usage ==
 +
<source lang='bash'>
 +
</source>
 
= WikiNuke =
 
= WikiNuke =
 
wikinukes.py allows mass deletion of pages
 
wikinukes.py allows mass deletion of pages

Revision as of 07:55, 18 December 2020

Click here to comment

What is it

Extended functionality for

  1. pywikibot
  2. mwclient

Github

Installation

via pip

pip install py-3rdparty-mediawiki
# alternatively if your pip is not a python3 pip
pip3 install py-3rdparty-mediawiki

upgrade

pip install py-3rdparty-mediawiki -U
# alternatively if your pip is not a python3 pip
pip3 install py-3rdparty-mediawiki -U

Via Source code

git clone https://github.com/WolfgangFahl/py-3rdparty-mediawiki
./install

wikipush / wikibackup / wikiedit / wiknuke / wikupload / wikiuser scripts

Setup method

If you installed with the method above console_script will have been added to your environment. You can e.g. check 

which wikipush
/Users/wf/Library/Python/3.8/bin/wikipush

and there should be a wikipush script in your path.

Linked Script method

The scripts for this method are in the scripts directory. They should be linked with the needed names 

ln wikipush wikibackup
ln wikipush wikiuser
...

and be added to your path e.g by doing a symbolic link 

cd $HOME/bin
ln -s $HOME/Documents/pyworkspace/py-3rdparty-mediawiki/scripts/wikipush .
ln -s $HOME/Documents/pyworkspace/py-3rdparty-mediawiki/scripts/wikiuser .
...

shared script

This script is the base script called "wikipush" which can be used using different names by hard linking as outlined above 

#!/bin/bash
# WF 2020-10-31
# wrapper for wikipush python
script=$(python -c "import os;import sys;print (os.path.realpath(sys.argv[1]))" $BASH_SOURCE)
scriptname=$(basename $script)
scriptdir=$(dirname $script)
base=$scriptdir/..
export PYTHONPATH="${PYTHONPATH}:$base"
case $scriptname in
   "wikipush")
	python -m wikibot.wikipush "$@"
	;;
   "wikiedit")
   	python -m wikibot.wikiedit "$@"
   	;;
   "wikiuser")
	python -m wikibot.wikiuser "$@"
	;;
   "wikinuke")
   	python -m wikibot.wikinuke "$@"
   	;;
   "wikiupload")
   	python -m wikibot.wikiupload "$@"
   	;;
   *)
	echo "undefined wikipush script behavior:  $scriptname"
	;;
esac

WikiPush

WikiPush allows to copy pages from one wiki to another including the images on the page. To identify yourself you use the credential property files created with the wikiuser script (using python) or the Mediawiki-Japi CommandLine

usage

wikipush -h
family and mylang are not set.
Defaulting to family='test' and mylang='test'.
usage: wikipush.py [-h] [-d] [-V] [-l] [-f] [-i] [-q QUERY] -s SOURCE -t
                   TARGET [-p PAGES [PAGES ...]]

Created on 2020-10-29

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -l, --login           login to source wiki for access permission
  -f, --force           force to overwrite existing pages
  -i, --ignore          ignore upload warnings e.g. duplicate images
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -s SOURCE, --source SOURCE
                        source wiki id
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

Example

wikipush -s smw -t test2 -q "[[Category:City]]|limit=5"
family and mylang are not set.
Defaulting to family='test' and mylang='test'.
copying 4 pages from smw to test2
copying Demo:Tokyo ...✅
copying image File:SMW-Info-button.png ...✅
copying image File:Tokyo-Tsukishima-0011.jpg ...✅
copying Vienna ...✅
copying Warsaw ...✅
copying image File:6140285934 02e81b845f z.jpg ...✅
copying Demo:Würzburg ...✅

wikibackup

usage

WikiNuke

wikinukes.py allows mass deletion of pages

usage

usage: wikinuke.py [-h] [-d] [-V] [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET [-p PAGES [PAGES ...]]

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -f, --force           force to delete pages - default is 'dry' run only listing pages
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

Example

The default behavior is a dry run only listing whether the pages exist 

wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3
deleting 3 pages in test (dry run)
1/3 (  33%): deleting deleteMe1 ...👍
2/3 (  67%): deleting deleteMe2 ...👍
3/3 ( 100%): deleting deleteMe3 ...👍

After checking you might want to (carefully) use the "-f" option to actually force the deletion: 

wikinuke -t test -p deleteMe1 deleteMe2 deleteMe3 -f
deleting 3 pages in test (forced)
1/3 (  33%): deleting deleteMe1 ...✅
2/3 (  67%): deleting deleteMe2 ...✅
3/3 ( 100%): deleting deleteMe3 ...✅

WikiEdit

wikiedit.py allows mass editing of pages using python regular expressions

usage

wikiedit -h
usage: wikiedit.py [-h] [-d] [-V] --search SEARCH --replace REPLACE [-f] [-q QUERY] [-qf QUERYFIELD] -t TARGET
                   [-p PAGES [PAGES ...]]

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  --search SEARCH       search pattern
  --replace REPLACE     replace pattern
  -f, --force           force to edit pages - default is 'dry' run only listing pages
  -q QUERY, --query QUERY
                        select pages with given SMW ask query
  -qf QUERYFIELD, --queryField QUERYFIELD
                        query result field which contains page
  -t TARGET, --target TARGET
                        target wiki id
  -p PAGES [PAGES ...], --pages PAGES [PAGES ...]
                        list of page Titles to be pushed

example

wikiedit -t test -q "[[isA::CFP]]"  --search "CALL FOR PAPER" --replace "CFP"
editing 1 pages in test (dry run)
1/1 ( 100%): editing CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) ...👍 |isA=CFP
-|Acronym=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) 
-|Title=CALL FOR PAPER Journal: Advances in Multimedia - An International Journal (AMIJ) 
+|Acronym=CFP Journal: Advances in Multimedia - An International Journal (AMIJ) 
+|Title=CFP Journal: Advances in Multimedia - An International Journal (AMIJ) 
 |Start date=2010/11/01
 }}
-CALL FOR PAPER
+CFP
 Journal: Advances in Multimedia - An International Journal (AMIJ)

wikiupload

wikiupload.py allows to mass upload files

usage

wikiupload -h
usage: wikiupload.py [-h] [-d] [-V] --files FILES [FILES ...] [-f] -t TARGET

Created on 2020-11-12

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  --files FILES [FILES ...]
                        list of files to be uploaded
  -f, --force           force to (re)upload existing files - default is false
  -t TARGET, --target TARGET
                        target wiki id

example

wikiupload -t test --files car.png
uploading 1 files to test
1/1 ( 100%): uploading car.png ...✅

wikiuser

wikiuser.py creates credential files and assigns a WikiId under which you can now operate. This simplifies access to your wiki. The credential file is compatible to the Java Mediawiki-Japi see CommandLine#Credential_mode

usage

wikiuser -h
usage: wikiuser.py [-h] [-d] [-V] [-e EMAIL] [-f FILEPATH] [-l URL]
                   [-s SCRIPTPATH] [-p PASSWORD] [-u USER] [-v VERSION]
                   [-w WIKIID] [-y]

WikiUser credential handling

  Created by Wolfgang Fahl on 2020-10-31.
  Copyright 2020 Wolfgang Fahl. All rights reserved.

  Licensed under the Apache License 2.0
  http://www.apache.org/licenses/LICENSE-2.0

  Distributed on an "AS IS" basis without warranties
  or conditions of any kind, either express or implied.

USAGE

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           set debug level [default: None]
  -V, --version         show program's version number and exit
  -e EMAIL, --email EMAIL
                        email of the user
  -f FILEPATH, --file FILEPATH
                        ini-file path
  -l URL, --url URL     url of the wiki
  -s SCRIPTPATH, --scriptPath SCRIPTPATH
                        script path
  -p PASSWORD, --password PASSWORD
                        password
  -u USER, --user USER  os user id
  -v VERSION, --wikiVersion VERSION
                        version of the wiki
  -w WIKIID, --wikiId WIKIID
                        wiki Id
  -y, --yes             immediately store without asking

Example

E.g. if you have an account on www.semantic-mediawiki.org you can start wikiuser in interactive mode. 

wikiuser
email: john@doe.com
scriptPath: /w
user: jd
url: http://www.semantic-mediawiki.org
version: Mediawiki 1.33
wikiId: smw
password: *****
shall i store jd smw? yes/no y/ny

Now you can e.g. use "smw" as the wikiid for this wiki when using wikipush

Prerequisites

You might want to prepare some credential ini files with the wikiuser script or Mediawiki-Japi CommandLine.

user-config.py

pywikibot expects a user-config.py file. The minimum recommended file for intranet usecases is: 

# https://stackoverflow.com/a/60885381/1497139
# Slow down the robot such that it never makes a second page edit within
# 'put_throttle' seconds.
put_throttle = 0
# avoid warnings ...
family='bitplan'
mylang='en'

The easiest way is to put it at $HOME/.pywikibot/user-config.py

Features

Encrypted credential handling

Py-3rdparty-mediawiki allows using pywikibot by simply giving each wiki an id and using the credential information created by MediaWiki-Japi. The needed family file is automatically created and registered. If you'd like to get a pure python solution for credential handling please file an issue on github - it's no big deal but i personally don't need it yet since i'm fine with the new CommandLine feature added recently.

Semantic MediaWiki API support

see https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/1

Example

from wikibot.wikibot import WikiBot
wikibot=WikiBot.ofWikiId("test2")
wikibot.site ...
Retrieved from "http://mediawiki-japi.bitplan.com/index.php?title=Py-3rdparty-mediawiki&oldid=1237"